Running a practice
Evaluating Your Practice
Gerard Gill, FRACGP, DRACOG, MAE, is in general practice in Launceston Tasmania. He is currently Vice Chairman of the Tasmanian Faculty and chairs the Tasmanian Research Committee. In 1996 he completed a GPEP sponsored Masters degree in Applied Epidemiology in General Practice Evaluation at the ANU.
In this series you have learnt about research as it applies to researchers in general practice. You have seen how a research question is defined, examined how to search the medical literature to discover what is known about the topic, seen the use of qualitative and quantitative approaches to data collection and had explained to you some of the problems of data collection and analysis.
As an active GP you may have found the topics interesting, but who has time for research? This final article in the series is titled 'Evaluating your own practice'. Every day you carry out this activity. Simple items like how long do your patients wait, or is there enough money earned to cover expenses? These are things you have been evaluating for a long time. You are an expert at evaluating many aspects of your practice. What I want to try and do in this brief article is to show you how knowledge of research approaches can help you by making it easier to carry out an evaluation and arrive at more accurate conclusions.
What is evaluation?
Evaluation in Australia has been defined under the General Practice Strategy as something other than research. Simply put, evaluating your practice is to compare your practice's performance in some area of activity with a benchmark. That benchmark could be a national standard such as a target for cervical cytology in women aged 20-70 years, a comparison with other practices such as the RACGP Interpractice Comparison, or it could be a comparison of your current practice performance with that in your practice at a previous point in time. I personally feel that evaluation is a form of applied research which requires a working knowledge of research methodology and tools.
Practice evaluation: is it for you?
You are the best person to evaluate your practice. You know what the problems are and have good knowledge of the capabilities that exist to carry out an evaluation. Getting your practice right is important to you, both because you are a professional who expects to deliver high quality care to your patients and because if your practice is performing poorly it could seriously impact on the lives of yourself, your staff and associated families. While currently you may not be the best person to carry out the evaluation, because you cannot give it the time (or may be unable to carry out such activities in an unbiased way) you must at least be involved in designing the evaluation study. If the aspect of your practice you wish to evaluate involves the performance of other doctors or practice staff, not involving them in the decision making processes when the evaluation is planned, may make it difficult to maintain their involvement. You would do well to ask a few patients their opinion about your plan if the evaluation involves aspects of patient behaviour (eg. how do you make delays for pre-booked consultation more acceptable).
Aspects of your practice to evaluate
Donebedian, an American writer on medical quality talks about quality in health care as examinable using the concepts of structure, process and outcome.1 Using Donebedian's framework in relation to your practice's financial functions provides a good example of how to approach evaluation.
Structure
This refers to the systems you have in place to deal with aspects for running the practice. If you bulk bill all your patients or are a cash only practice, you can get by very easily with a manual accounting system. If you have multiple surgeries, issue accounts to all your patients and have several categories of fee level, your accounting needs may be better handled by a computerised accounting system. The structure of your practice will determine how you may best deal with patient accounts.
Process
Process refers to how the structures you have in place function. Let us assume you are bulk billing all your patient contacts. Process issues would include items such as:
- do you get vouchers signed for all your patients?
- are the vouchers correctly filled in?
- do they sit around for some weeks to months before being dispatched to Medicare?
Outcome
Outcome refers to what happens after an event occurs. In financial terms this is the amount of money you take home from the practice. In other areas of practice performance, outcomes may be more difficult to measure readily. A good example is the result from mammography take up by your patients. If your outcome measure is death and it takes 12 years of screening of 1000 women to prevent one cancer death,2 you will have to wait several lifetimes to see the effect of mammography on the death rates of your patients from carcinoma of the breast. Often surrogate outcomes are used such as the proportion of eligible women who have mammography performed or the blood pressure recorded on those you treat for hypertension. These are assumed to be linked to the ultimate outcome such as death in the case of mammography or cardiovascular complications in the case of hypertension.
Planning an evaluation
Planning an evaluation is a bit like planning a research project. At the start you define the aspect of your practice you want to evaluate. A decision will be made as to whether the evaluation is of structure, process or outcome of the aspect being considered. You explore how others might have examined this aspect or how others might attempt to do so. Searching the literature or asking experts may give you some direction (Table 1). This exploration may also turn up research tools (called instruments) that you might use to measure that aspect. A published instrument is usually something like a questionnaire which has been shown to be reliable (when administered again close to the original attempt to the same person you get similar answers) and valid (measures what you want it to measure). A good example of an instrument used in general practice evaluation is the Patient Satisfaction Survey produced by the RACGP South Australian Health Promotion and Research Unit.
Table 1. Planning an evaluation
- Resource personnel
- University academic GPs
- Project officers in local GP division
- RACGP research network
- Local faculty research committee
- Colleagues in other practices
For some evaluations such as our earlier example of financial aspects, collecting data like simple figures will do.
It is important that the data you collect are accurate and actually represent what is happening. You must compare apples to apples and not apples to oranges. For example, comparing income over 2 years must take into account such items as severe influenza outbreak or the effect of any unusual one off expenditure. You need to consider how easy the data will be to collect, who will collect it and how you will confirm it is being collected without any serious omissions or errors.
Then you need to think how you are going to analyse it and what the findings of your analysis mean. If your analysis involves statistics it is wise to seek some advice on which test is best to use. It is important to know the limitations of statistics. For example, you are looking at the outcome of the recorded systolic blood pressure for those in your practice who are on therapy for hypertension after you have attempted to improve your performance at lowering blood pressure. You may now have an average systolic blood pressure which is 2 mmHg lower than before you lifted your game. This result may be highly statistically significant but will make little difference to the outcome of the number of patients developing cardiovascular disease. If the difference was 5 mmHg reduction in systolic blood pressure we might expect a larger reduction in the number of patients later developing cardiovascular disease. Finally, you will need to consider from your evaluation what needs changing to improve your practice performance. This is probably the most difficult step. You may need to change aspects of your behaviour or of other practice doctors and staff. However, you are in the business of trying to change people's behaviour all the time. The approaches you use in say, smoking cessation, may be helpful to identify what stage people are at in their awareness of the need to change, are they committed to changing, or are they changing? You have the skills to move them from one phase to another, to recognise when they are not ready to change, and to support them while they change. Those of you who are familiar with the concept of the audit cycle may see the four phases nicely woven into my portrayal of evaluation. Audit is in essence just a more formal structured evaluation which is repeated.
Conclusion
Practice evaluation is part of general practice's core business. You have the knowledge and skills to plan for its use in your practice and you already do it on some level. Preparation makes it a lot easier to do. I should add that it is actually fun.
Suggested reading
Wadsworth Y. Everyday evaluation on the run. Melbourne: Action Research Issues Association (Inc), 1990. (Perhaps more social work orientated but free of a lot of jargon and very practical).
References