Contents
18.1 Putting Outcome Measures into the Correct Perspective 535 18.1.1 Introduction 535
18.1.2 Definition 535
18.1.3 Putting the Role of Outcome Measures into the Correct Perspective 535 18.1.4 Key Concept 1 535
18.1.5 Key Concept 2 535
18.1.6 Recapitulating Elements of a Good Study Design 536
18.1.7 Reason for the Recent Surge in Interest in Outcome Measures 536 18.1.8 Limitations in the Use of Outcome Measures in Rehabilitation 536 18.1.9 Why the Recent Surge in Interest in Quality
of Life Outcome Measures? 537
18.2 Selecting the Appropriate Outcome Measure for Your Research 537 18.2.1 Key Elements of Any Outcome Measure 537
18.2.2 Using Selection of Clinical Investigations to Ease Understanding 537 18.2.2.1 The First Question 538
18.2.2.2 The Second Question 538 18.2.2.3 The Third Question 538 18.2.2.4 The Fourth Question 539
18.2.3 Summary of Key Considerations in Choosing Outcome Measures 539 18.2.4 Other Myths Concerning Outcome Measures 539
18.2.5 Caution in the Use of Outcome Measures 540 18.2.6 General Conclusion 540
18.3 Use of Outcome Measures in the Field of Rehabilitation 540 18.3.1 Pitfall of Using Outcome Measures in the Field of Rehabilitation 540 18.3.2 Point of Note 540
18.4 Recent Trends 541
18.4.1 What Are Some Recent Trends in the Use of Outcome Measures? 541 18.4.2 Use of Patient-Based Subjective Outcome Measures 541
18.4.3 Chief Argument for Using Subjective Patient-Based Measures 541 18.4.4 Author’s View 541
18.5 Other Areas of Interest 542 18.5.1 The Question of “Validation” 542
18.5.2 Elaborating the Concept of “Validation” 542 18.5.3 Question of “Specificity” 542
18.5.4 Learning Points Concerning the Use of Questionnaires 543
Outcome Measures
and Clinical Governance
18
18.6 Clinical Governance 543
18.6.1 Concept of “Clinical Governance” 543 18.6.2 Key Elements of Clinical Governance 543
18.6.3 Concluding Remarks: Is There an Ideal Outcome Measure? 543 General Bibliography 544
Selected Bibliography of Journal Articles 544
18.1 Putting Outcome Measures into the Correct Perspective
18.1.1 Introduction
n The reader will notice that there are more and more books on the market on the topic of outcome measures; we will consider the reason for the boom in interest in the coming discussion
n But the most important thing to point out is that seldom, if ever, do books on outcome measures highlight the fact that the design of our re- search is much more important than which of the dozens of outcome measures for a given pathology we will pick for our research study
n As the recommended outcome measures for the orthopaedic condi- tions discussed in this book were given in their respective chapters, this chapter will be short and will mainly concentrate on some basic principles
18.1.2 Definition
n “Outcome” is defined as a change in a state or situation that arises as a result of some process of intervention
n “Measure” refers to the quantification of data in some way, either in absolute or relative terms
18.1.3 Putting the Role of Outcome Measures into the Correct Perspective
n The discussion that follows will hopefully put the role of the different outcome measures (the numbers of which are increasing by the min- ute) into the correct perspective
18.1.4 Key Concept 1
n Although knowledge of what constitutes a good outcome measure is im- portant, it is even more essential that the fundamental design of the re- search study or audit we are going to embark on is properly designed
18.1.5 Key Concept 2
n Outcome measures can be likened to surgical tools that the surgeon uses
n No matter how good the surgical tools we have, they cannot substitute for poor surgical technique
a 18.1 Putting Outcome Measures into the Correct Perspective 535
n Similarly, no matter how good an outcome measure the researcher picks, it cannot overcome the deficiency of a poorly designed study
n On the other hand, reasonably reliable study results that may be worthy of publication can sometimes be obtained by a very well-de- signed study even in the face of outcome measures of slightly poorer quality
18.1.6 Recapitulating Elements of a Good Study Design
n Clear statement of the research goal
n Elimination of bias, e.g. use of controls, single or double blind studies preferred
n Clear statement of the level of evidence
n Proper power studies before research begins to ensure the magnitude of sample size required
n Selection of proper assessment and outcome tools appropriate to the pathology at hand, as well as with due regard to the goal of the study
n Proper presentation and analysis of data
n Proper selection and use of statistical analyses
n Appropriate discussion and conclusion based on the findings of the study
18.1.7 Reason for the Recent Surge in Interest in Outcome Measures
n The main reason comes from aspects of clinical governance and clini- cal audit, which will be discussed separately. Often, health authorities are reluctant to pay for services rendered unless supported by “good outcome”; an example is seen in “seating clinics” discussed in Chap. 6
n Moreover, when it comes to rehabilitation, many health administrators still have the misconception that a subspeciality like rehabilitation can be easily assessed by the use of outcome measures
18.1.8 Limitations in the Use of Outcome Measures in Rehabilitation
n The outcome of the rehabilitation service as a whole may differ from the outcome arising from any single component of the service, re- membering that the rehabilitation process is a multidisciplinary pro- cess with multiple interventions
n Often, the final outcome can be influenced by factors the team does not have control over, e.g. community resources, level of unemploy- ment (Clin Rehabil 2001)
n As pointed out by ICF of WHO, besides the above-mentioned social context factors and physical factors of the patient (such as the num- ber of impairments), there are factors pertaining to “personal con- text”, e.g. attitudes of family, patient’s beliefs and expectations
18.1.9 Why the Recent Surge in Interest in Quality of Life Outcome Measures?
n Increased realisation that functional status rating tools that measure consumers’ performance in activities that are meaningful to them al- lows service providers to detect changes in functional status based on perceived quality of life
18.2 Selecting the Appropriate Outcome Measure for Your Research
18.2.1 Key Elements of Any Outcome Measure
n The numerous textbooks on outcome measures are usually rather dogmatic: they just tell you what the key elements are without ex- plaining why. Then follows a list of validated and non-validated out- come measures for common clinical conditions
n In view of the above, we will use an example of a clinician selecting an investigation into a clinical problem as illustration. In this way, it will be much easier to understand and demystify aspects of outcome measures
18.2.2 Using Selection of Clinical Investigations to Ease Understanding
n I believe the readers are mostly clinicians, and we order clinical inves- tigations every day
n What, then, makes us choose a particular clinical investigation of a patient?
a 18.2 Selecting the Appropriate Outcome Measure for Your Research 537
18.2.2.1 The First Question
n We first ask ourselves what is the aim of ordering an investigation. In this context, we will use a 4-week postoperative TKR patient with re- currence of knee pain as our example
n In this example, suppose the surgeon wants to rule out subacute sep- sis after examining the patient, and this is the chief aim in mind
18.2.2.1.1 Corollary
n Similarly, in the case of outcome measures we must first have the aim of our clinical research before proceeding to choosing an outcome measure
18.2.2.2 The Second Question
n The second question we ask ourselves is what determines our choice if there is more than one possible investigation to choose from
n In this case, our options include: ESR, C-reactive protein, diagnostic knee tapping, bone scan, open biopsy, frozen section, etc. The list is endless
18.2.2.2.1 Corollary
n This scenario is rather like the situation in which there are many out- come measures for the same orthopaedic condition we want to study
18.2.2.3 The Third Question
n Given the options, what determines our choices?
– Depends on the test sensitivity, specificity and reliability
– We will remind ourselves exactly what we are looking for in each test, i.e. the content
– We need to interpret the test results
– We need to decide whether to do one or more such investigations in our patient
– If all tests are negative, but we still suspect infection, is there a test that is sensitive to changes in the patient’s condition?
18.2.2.3.1 Corollary
n Similarly, in choosing outcome measures for our study:
– We need to reveal the content or in other words, what exactly is a given outcome measure looking at
– We will also come across cases in which more than one outcome study is needed, e.g. including both subjective and objective types
n We need to know how to interpret the meaning of the scoring used in different outcome measures
n We need to know how sensitive our test is – in the field of outcome measures we choose to call this “validity”
n We need to know the reliability or inter- and intra-observer reliability
n We need to know how responsive the test(s) we selected is/are well enough to pick up subsequent changes in the patient’s condition (for better or for worse)
18.2.2.4 The Fourth Question
n Finally, we need to sign the consent and the patient invariably will ask questions like: “Is the procedure going to be very painful? Will I be incapacitated for days?” – in short, patient’s acceptability
n On the other hand, we know at the back of our minds how simple or how complex the procedure is – in short, whether it is a technically demanding procedure
18.2.2.4.1 Corollary
n Similarly, in the case of outcome measures, whether it is user-friendly on the part of the patient and surgeon needs to be considered
18.2.3 Summary of Key Considerations in Choosing Outcome Measures
n Review the aim of our study and nature of the pathology and study population
n Reliability
n Content validity and content interpretation
n Responsiveness to change
n Whether user-friendly to both the patient and the surgeon
18.2.4 Other Myths Concerning Outcome Measures
n Misconception that better outcome measure will automatically equate with better function
n In fact, it is interesting to note that the yardstick of what constitutes a good outcome measure differs widely in the eyes of different people:
some prefer simple measures, others prefer more disease-specific, yet others prefer more generic ones
a 18.2 Selecting the Appropriate Outcome Measure for Your Research 539
18.2.5 Caution in the Use of Outcome Measures
n The measure chosen should only focus on the intended area of inter- est, not on extraneous factors, i.e. do not get side-tracked
n The measure chosen should preferably have been validated previously for the measurement of the item we are studying. Example: use of GMFM-66 as a monitor of muscle strength in CP children throughout his/her rehabilitation process
18.2.6 General Conclusion
n Outcome measures are only tools, not solutions
n All parties must remember that the validity of conclusions of any study depends mainly on the study design and logic of the study
n Using good and appropriate outcome measures increases the chance of success of the study, but cannot compensate for bad study design
18.3 Use of Outcome Measures in the Field of Rehabilitation
18.3.1 Pitfall of Using Outcome Measures in the Field of Rehabilitation
n Rehabilitation involves a multidisciplinary team process, and the pa- tient receives multiple interventions going through the process
n The desired outcome of the service is affected and constrained by many factors outside the team’s control; hence, application of clinical governance in rehabilitation is not straight forward
n Since monitoring outcome may be an ineffective way of assessing (Mant et al., BMJ 1995), especially when it comes to rehabilitation, an alternative way is to monitor “adverse outcomes” such as falls leading to fractures, pressure sores, etc.
18.3.2 Point of Note
n The field of rehabilitation itself involves periodic re-evaluation of its process. Re-evaluation is a main component of rehabilitation besides the three other components; namely, assessment, goal setting and in- tervention
n Most will agree a most convenient and appropriate means of monitor- ing is to review patient documentation
n Occasionally, use of a standard outcome measure is appropriate in a rehabilitation service dedicated mainly to the treatment of a major clinical condition, e.g. geriatric hip fractures
18.4 Recent Trends
18.4.1 What Are Some Recent Trends in the Use of Outcome Measures?
n Many orthopaedic surgeons are accustomed to using more objective outcome measures, e.g. Knee Society scores in patients undergoing TKR
n There is recently a trend towards an increase in the use of subjective outcome measures, not only in orthopaedics, but also in other fields of medicine
18.4.2 Use of Patient-Based Subjective Outcome Measures
n These are definitely increasing in popularity
n The reader will note that more and more peer reviewed papers have now included patient-based subjective outcome measures
18.4.3 Chief Argument
for Using Subjective Patient-Based Measures
n Particularly after operation, e.g. spinal fusion, while the X-ray may look perfect and the attending surgeon contented, the patient may still be dissatisfied, say, because of persistent pain
n Patient satisfaction is becoming a very important subjectively based outcome measure, particularly after surgical procedure
n Subjective measures like the use of questionnaires frequently correlate well with the results of clinical assessment of health status and work ability (Eskelinen et al., Scand J Work Environ Health 1991)
18.4.4 Author’s View
n In most cases, the use of subjective and use of objective outcome measures compliment each other
n It is the author’s view that many research studies require the conco- mitant use of both objective and subjective outcome measures
a 18.4 Recent Trends 541
18.5 Other Areas of Interest
18.5.1 The Question of “Validation”
n To better illustrate the concept of validation, we will use “Validation of Questionnaires” as an example
18.5.2 Elaborating the Concept of “Validation”
n Validation of a questionnaire involves testing its test-retest reliability (reproducibility), responsiveness (ability to detect clinically important change), and validity
n Face validity is the concept that questions are relevant. Content valid- ity is determined by the consensus of experts. Construct validity is determined by correlating participants’ answers to the questions with objective measurements
n However, some experts feel that there is no accepted standard of what constitutes validation. Validation is self-proclaimed, usually after a study has been published in a medical journal (according to Sarins)
18.5.3 Question of “Specificity”
n Use of outcome measures that are specifically useful for the condition at hand should be borne in mind
n For example, if we intend to assess the subjective outcome of ACL-in- jured patients, we should use measures like ACL-QOL (quality of life) questionnaire, rather than, say, the WOMAC or SF-36
n The Short Form-36 (SF-36) is a generic measure that includes ques- tions about general health, activities performed, problems at work, emotional issues, physical activities, pain and personal feelings. One of the 36 questions (specifically, a question regarding the level of vig- orous activities) could relate to knee instability. The SF-36 has been validated for quality of life, but not for knee problems
n The Western Ontario and McMaster Universities (WOMAC) Osteoar- thritis Index was developed to assess patients who have osteoarthritis of the hip and/or knee. The index consists of 24 questions related to pain (5), stiffness (2) and physical function (17). The response to each question is scored from 0 to 4. The maximum score is 96. None of the questions are about instability or sports participation
– A patient who has a tear of the ACL, causing the knee to give way with pivoting motions and resulting in positive Lachman and pi-
vot-shift tests, can still receive a score of 96. The WOMAC scale has been validated for osteoarthritis of the knee, but not for knee instability
n The ACL-QOL questionnaire was developed, pretested and validated for patients who have a torn ACL
18.5.4 Learning Points Concerning the Use of Questionnaires
n The more specific a questionnaire, the more sensitive it will be for discriminating outcomes between patients who have the disorder
n A one-time use of a questionnaire on the status of a condition is not a “measure of outcome”. The answers to the questions are merely de- scriptions of subjective symptoms at a single point in time. Compar- ing responses to the same questions asked before and after interven- tion are subjective measures of outcome
18.6 Clinical Governance
18.6.1 Concept of “Clinical Governance”
n The modern definition of clinical governance in the eyes of health authorities should involve the process of monitoring and improving the quality of clinical services. Also, since the healthcare organisation needs to take responsibility for its own affairs, it tries to locate
“power and responsibility” at a point within the system (usually by nominated individuals) to take full responsibility for the delivery of a quality service
18.6.2 Key Elements of Clinical Governance
n Reach an agreed definition of the quality aimed at
n Agree on who is responsible for achieving and maintaining the agreed standards
n Agree on how quality is to be measured
18.6.3 Concluding Remarks: Is There an Ideal Outcome Measure?
n There is no one measure that is ideal
n It is the author’s view that many studies require the concomitant use of both objective and subjective outcome measures
a 18.6 Clinical Governance 543
n Outcome measures that are validated to the disorder we are studying are concise and easy to administer by team members and evaluated serial functional changes will be given priority
n As far as rehabilitation is concerned, since a multidisciplinary team is involved; one should understand that the outcome of the rehabilita- tion service as a whole may differ from the outcome arising from any single component of the service
n Only proper understanding of the principles and uses of outcome measures will pave the way to satisfying the basic elements of clinical governance
General Bibliography
Pynsent P (2004) Outcome measures in Orthopaedics and Orthopaedic Trauma. Oxford University Press, UK
Selected Bibliography of Journal Articles
1. Mant J, Hicks N (1995) Detecting differences in quality of care: the sensitivity of mea- sures of process and outcome in treating myocardial infarction. BMJ 311:793–796 2. Tuomi K, Eskelinen L et al. (1991) Effect of retirement on health and work ability
among municipal employees. Scand J Work Environ Health, 17(Suppl 1):75–81
