Evidence-Based Practice in Communication Disorders: An Introduction

1. Independent confirmation and converging evidence

It is extremely rare for a single study to provide the definitive answer to a scientific or clinical question, but a body of evidence comprising high quality investigations can be synthesized to approach a definitive answer even when, as is likely, results vary across studies. When the question concerns treatment efficacy, the highest evidence ranking goes to well-designed meta-analyses that summarize results across a number of scientifically rigorous studies. In many cases, results are expressed using both summary statistics and a graphic representation of the direction, size and precision of findings from individual studies. This level of evidence remains relatively rare even in medicine, but a growing number of studies of treatment efficacy are eligible for meta-analysis and meta-analyses are beginning to appear in the communication disorders literature (e.g., Casby, 2001; Robey, 1998). A number of organizations sponsor reviews of evidence according to explicit and stringent criteria; these include the U. S. Department of Health and Human Service's Agency for Healthcare Research and Quality (http://www.ahrq.gov), the Cochrane Collaboration (www.cochrane.org), and the Scottish Intercollegiate Guideline Network (www.sign.ac.uk). A single meta-analysis or systematic review of evidence may not yield results that are so uniform as to preclude disagreement and debate, especially if the number of high quality studies available for inclusion is relatively small. However, the principle of seeking converging evidence from multiple strong studies is inextricably linked to the EBP orientation.

Return to Top

2. Experimental control

The design features of individual studies also influence ratings of evidence quality. In the EBP framework, evidence from studies that are controlled (i.e., that contrast an experimental group with a control group) and that employ prospective designs (in which patients are recruited and assigned to conditions before the study begins) is rated more highly than evidence from retrospective studies in which previously collected data are analyzed, because the reliability and accuracy of many measures are difficult or impossible to ensure post hoc. In addition, group comparison studies are rated more highly when patients are randomly assigned to groups than when they are not, because random assignment reduces the chance that groups might differ systematically in some unanticipated or unrecognized ways other than the experimental factor being investigated.

Lower evidence ratings generally are assigned to quasi-experimental studies, including cohort studies in which patients with and without a variable of interest are followed forward in time to compare their outcomes, and case-control designs in which patients with and without an outcome are identified and compared for their previous exposure to a variable of interest. Evidence from quasi-experimental studies ranks lower than evidence from controlled studies because only through random assignment can the risk of differences due to unknown biases be minimized. Evidence from nonexperimental designs such as correlational studies, case studies (N = 1), and case series is rated even lower due to the lack of a control group, but even evidence from nonexperimental study designs outranks statements of belief and opinion in EBP rating schemes.

It is worth emphasizing that experimental control is only one of the themes important to rating evidence quality; regardless of its design, every study addressing a clinical question also must be evaluated with respect to such factors as the potential for subjectivity and bias, the importance of its results, and its relevance and feasibility. Especially when they are designed so as to maximize experimental control and to minimize bias, quasi- and nonexperimental studies can provide evidence that is crucially important to the early stages of investigation into a phenomenon and can lay the necessary groundwork for studies with larger samples, random assignment, and strict experimental control. For example, investigators can provide some evidence of experimental control in single-subject studies by comparing treated and control goals in a multiple-baseline design or by randomly assigning treatment and control conditions to different time periods in a multiple crossover or alternating treatments design. In addition, as noted by Sackett et al. (2000), well-designed single-subject studies can be extremely helpful in assessing the effectiveness of treatment for an individual patient. Thus, carefully conducted single-subject studies should be recognized as having an important role to play in EBP although their results will always require confirmation via stronger designs.

Return to Top

3. Avoidance of subjectivity and bias

An important criterion for credible evidence is that observers, investigators, statisticians, others involved with patients, and if possible the patients themselves, be kept unaware of information that could potentially influence, or bias, the results of a study. This tactic is known as blinding, concealment, or masking. Blinding addresses a particular threat to the validity of patient-oriented evidence: the seemingly inescapable bias that clinicians have toward believing that their efforts are beneficial. George Pickering (1964; cited in Barrett Connor, 2002) observed that belief in the value of one's efforts is a pre-requisite to clinical practice, but such belief is at odds with the objectivity that is fundamental to the scientific method (Meehl, 1997). There is persuasive empirical evidence of the need for blinding in studies of medical treatment; one analysis showed that estimates of treatment effects from studies without blinding were substantially larger than those from studies in which treatment conditions were concealed (Schulz, Chalmers, Hayes, & Altman, 1995). Observer expectations have been shown to influence even such seemingly objective measurements as recording fetal heart rates from monitors (Sackett et al., 1991). The fact that relatively few studies in speech-language pathology and audiology employ strategies to ensure adequate blinding may be one reason literature on communication disorders is underrepresented in evidence-based reviews. Complete blinding of patients and clinicians may be impossible in some studies, especially for behavioral treatments for which a placebo condition cannot be constructed. However, even in such studies a number of steps can be taken to minimize the potential for bias, such as ensuring that treatment effects (positive or negative) are measured not by the clinician, the investigator, or a family member but rather by independent examiners who rate patients without knowing their treatment assignments. Similarly, examiners can rate unlabeled, randomly ordered recordings from different stages in the course of intervention (e.g., pre-, intra- and post-treatment) to minimize the potential influence of their expectations about treatment effects.

Another important control for potential bias that influences evidence ratings is the requirement that outcomes be reported for every patient originally enrolled in a study, not just for the patients who complete it. This ensures that patients who did not complete the study as planned are taken into account in analyzing effects, avoiding the understandable tendency to focus only on patients who have positive outcomes. In randomized trials this approach, known as the “intention-to-treat” analysis, means that patients must be analyzed as part of the treatment group to which they were originally assigned even if they did not actually receive the treatment as planned (e.g., Moher, Schulz, Altman, et al. 2001; Sackett et al., 2000).

Return to Top

4. Effect sizes and confidence intervals

The EBP orientation emphasizes that studies of clinical questions should specify and justify the size of effect that is deemed clinically important and should provide evidence that statistical power is adequate to detecting an effect of this magnitude. Appreciation of the need to consider not just statistical significance (i.e., the probability that differences or effects were not chance events), but also practical significance (i.e., the magnitude of differences or effects, usually in the form of a standardized metric such as d or omega-squared) has been growing for at least 25 years, culminating in the mandate that information on effect sizes and statistical power be included in every published study (Wilkinson & APA Task Force on Statistical Inference, 1999). A variety of effect size indices exist (e.g., Huberty, 2002); according to Cohen (1990, p. 1310) the important point is to convey “… the magnitude of the phenomenon of interest appropriate to the research context.”

The EBP orientation also emphasizes the need for investigators to report the confidence interval (CI) associated with an experimental effect. CIs reflect the precision of the estimated difference or effect, specifying a range of values within which the “true” value is expected to occur with a given probability for a certain level of Type I error. Narrower CIs offer stronger (i.e., more precise and interpretable) evidence than wider CIs; studies in which samples are large and measurement error is small yield narrower CIs. This fact explains why, all else being equal, evidence from studies with large samples is likely to be ranked higher than evidence from studies involving smaller samples. It is increasingly common for investigators to provide CIs in published reports of their studies. Sackett et al. (2000; Appendix 1) provide a helpful review of the interpretation of CIs as well as procedures for calculating CIs for various types of diagnostic and treatment studies.

Return to Top

5. Relevance and feasibility

Relevance and feasibility are also considered frequently in rating the quality of patient-oriented evidence. Relevance of evidence is considered highest when the patients studied are typical of those commonly seen in clinical practice (Ebell, 1998), and/or when the clinical decision being studied is one that is difficult to make. Feasibility or applicability (Scottish Intercollegiate Guidelines Network, 2002) is high when the screening, diagnostic, or treatment activity being investigated is one that could reasonably be applied or used by practitioners in real-world settings. For example, some conditions can be diagnosed as accurately by interview as by time-consuming and expensive tests; the former would accordingly out-rank the latter on feasibility. It may not be possible to provide evidence of relevance and feasibility for results from studies at the early stages of investigation into a clinical question, but these factors must ultimately be included in evaluating the strength of evidence as a line of inquiry progresses.