Nature of the evidence base for evidence-based mental health care

Heather M. SCHLOSS & David A. F. HAAGA *
American University, USA

A distressed couple seeks therapy with the goals of reducing conflict, increasing their relationship satisfaction, and navigating their children’s teenage years with less stress. A socially anxious law student suffering panic attacks when called upon in class discussions wants help to reduce the frequency and severity of panic and increase her confidence in public speaking situations. An elderly man with a decades-long problem of compulsive hoarding is urged by his landlord to obtain help to reduce apartment clutter in order to stave off eviction for creating a fire hazard.

Correspondence concerning this article should be addressed to:

David A. F. Haaga

Department of Psychology, Asbury Building, American University, Washington, DC 20016-8062, USA

E-mail: dhaaga@american.edu

Preparation of this article was facilitated by grant 2 R15 CA77732-02-A1 from the National Cancer Institute.

In each of these scenarios, a responsible psychotherapist would need first to collect additional assessment information. Are there certain types of conflicts the couple has an especially hard time resolving, and how do they talk to one another about them? When did the panic attacks begin, and what symptoms and sensations are experienced? What is the subjective meaning of the “clutter” the man is hoarding, and what does he fear would happen if he discarded most of it?
In principle, there is always more that one would like to learn about a new patient, but at some point therapists must decide on at least a provisional treatment strategy, subject to later modification depending on treatment response. These decision points raise two interrelated concerns: (a) how to make the choice of treatment strategy, and (b) how to justify that choice to various interested parties. Interested parties include obviously the patient(s), but also perhaps the patient’s family, especially in the context of therapy for children or for cognitively impaired adults. In a litigious society, justifying one’s treatment strategies to the satisfaction of the legal system can become an issue if the patient’s health worsens. Finally, an increasingly prominent aspect of accountability in psychotherapy is the need to justify treatment approaches to third party payers for that treatment.
Although “evidence-based practice” is a relatively new term (Bruce & Sanderson, 2005), it has long been the case that most or all psychotherapists would expect to have to provide some sort of evidence when called upon to justify their treatment decisions. There is rarely an airtight deductive argument generating one form of psychotherapy as the only logical method to try. Moreover, argument by assertion (“because I said primal scream is the best therapy in all cases!”) is unlikely to prove convincing.
If choices must be made, and if there is widespread agreement that the choices should be predicated on evidence, then why is there so much controversy concerning evidence-based practice? We see the crux of these debates as being what forms of evidence will be treated as necessary, contributory, or sufficient to justify choice of treatment strategies.
One generally well-respected form of evidence consists of the outcomes of randomized clinical trials of a specific therapy approach. In principle, random assignment to conditions results in groups that are equal in all relevant ways before treatment begins. The groups are then treated in the same manner throughout the study in all respects but one – type of psychotherapy provided – and any resulting differences after treatment has ended can be confidently attributed to the independent variable of type of therapy.
In practice, randomized clinical trials of psychotherapy methods are exceedingly difficult to conduct in a manner that will yield unambiguous results (Haaga & Stiles, 2000). To take one example, therapists vary in their competence, both in general and with regard to specific forms of therapy. Measuring the competence of therapists’ performance independent of the difficulty level of a patient or the progress being made is not easy. Therefore, when results are poor, it can be problematic to determine whether the method being tried does not work or was not implemented properly. Perhaps in large part for this reason, reviews have shown considerable evidence of an “allegiance effect” in psychotherapy research, such that methods preferred by the principal investigator are especially likely to achieve favorable results (Luborsky et al., 1999). It seems likely that the less preferred methods are applied with a lesser degree of expertise.
A second general complication relating to interpreting randomized clinical trials in psychotherapy concerns external validity of the findings. In other words, to what extent can the results obtained within the experiment be extrapolated to the scenarios confronting the practitioner? If the effort to minimize within-group heterogeneity (e.g, by standardizing principal diagnosis, excluding highly comorbid cases or suicidal patients) results in an experimental group substantially unlike one’s normal caseload, then it is not certain that the results from such a study provide a good indication of what would happen in ordinary clinical practice.
Third, even if we can be confident that a particular system of therapy works reasonably well on average, it does not follow that all aspects of the treatment package are important, or that it works via the mechanisms hypothesized by its originators. For example, if Acceptance and Commitment Therapy (Hayes, 2004) reduces panic attacks more than does a no-treatment control or a treatment-as-usual condition, it does not necessarily mean that it does so by ways of increasing acceptance and distress tolerance.
Given these complications, it is small wonder that therapists have not immediately embraced the results of randomized clinical trials as having definitive practice implications. And yet, the APA Task Force on Promotion and Dissemination of Psychological Procedures charged with identifying empirically supported therapies regards RCT results as the single best source of empirical support (Chambless & Hollon, 1998).
Each of the articles under consideration in this review shares a point of departure – current methods of empirically identifying what works and why and how it works are subject to limitations as guides to clinical practice. Where they differ, in our reading, is in their implications for a constructive process of improvement, such that each generation of therapy research proves more helpful than the one before in generating new and reliable knowledge.

Levant (2004): Advocating Business as Usual

Levant (2004) notes the existence of controversy concerning efforts to promote use of, and training in, empirically supported treatments (ESTs) and concludes that “from the practitioner/educator’s perspective, the controversy seems to stem from the attempts of some clinical scientists to dominate the discourse on acceptable practice and impose very narrow views of both science and practice” (p. 220). Whether or not one side of an argument is attempting to dominate discourse seems like a subjective issue and irrelevant to whether their argument is correct. The characterization of views of science and practice as narrow calls for articulation of a broader view that could be seen as beneficial for enhancing clinical outcomes.
Levant (2004) goes on to acknowledge the existence of “shoddy practices in our field” but states that “However, every profession in every society throughout history has had to deal with those practitioners [who] do not fulfill their ethical and professional responsibilities” (p. 220). This exculpatory argument on behalf of therapists who ignore the empirical literature as a guide to practice has the same force as “all the other kids shoplift too” when one is caught stealing a candy bar. Crooked accountants do not relieve psychologists of the responsibility to make as good use of science as we can in helping patients.
Lamenting the negative effects of the empirically supported treatments movement on practitioners, Levant (2004) cites provision of ammunition for insurance companies wanting to restrict practice in order to reduce costs, influence on funding agencies to require empirically supported treatments, and possible future harm to practitioners in legal situations if empirically supported treatments are considered to represent standard of care. Accreditation of training programs and approval of CE sponsors on the basis of empirically supported therapies are also listed as possible hazards. None of these projected or real hazards is problematic, however, unless there is a better alternative to use in determining what methods should be reimbursed and taught.
Another of Levant’s arguments against the sufficiency of EST criteria is simply incorrect rather than arguable. He contends that a weakness of the EST lists is that many patients do not meet criteria for just one DSM diagnosis. However, EST criteria stipulate that client samples be well-defined [precisely so that one can tell whether the results are likely to apply to the patients a practitioner is trying to treat], not that they be defined by a single DSM diagnosis. One of us contributed a section of a task force report (Chambless et al., 1998) based upon treatments for a behavior (cigarette smoking), not the associated diagnosis (nicotine dependence), and this is perfectly consistent with EST logic.
In the end, Levant (2004) touts clinical judgment as the best basis for helping the unique individual clients seeking our services. He acknowledges that “some condemn clinical judgment as subjective” but responds that “clinical judgment is simply the sum total of the empirical and clinical experience and skill that clinicians bring to bear on the clinical enterprise. It is their duty to understand and treat the particular, unique person before them. Practitioners know that the unique variables are at least as important as, if not more important than, the common characteristics that enable diagnosis” (p. 222).
A difficulty with this position is that it does not take adequately into account the extensive literature documenting the fallibility of clinical prediction methods relative to actuarial schemes (Grove et al., 2000). Yes, everyone is unique, but our ability to judge in what manner standard protocols need to be adapted to that uniqueness is in doubt.
Perhaps more importantly, reliance on clinical judgment has a static quality. Unless we assume that experience alone sharpens our judgment and thereby our ability to effect positive outcomes [and the literature does not generally support this view; Christensen & Jacobson, 1994], the judgment of the individual clinician holds little promise as a source of progressive improvement of outcomes. Only when the principles underlying that judgment can be codified [as in a therapy manual] and implemented with high fidelity by others and tested in research can we hope to gain reliable knowledge of their validity. It is the self-corrective, progressive quality of scientific research fields that accounts for their success and cumulative character, not the brilliant insight of the master specialist alone.
Two other recent papers, though critical of the current database, make a stronger contribution than does Levant (2004) because of their attention to this very issue. Rather than simply point out problems with existing research literatures and fall back on clinical judgment, they make specific constructive suggestions for how research practices might be changed in order to promote more rapid progress toward a valid and useful psychotherapy research database. Doss (2004) achieves this aim by first examining current studies of “change” in psychotherapy and then by presenting a refined methodological framework for future investigations of the therapeutic process. Westen, Novotny, and Thompson-Brenner (2004) similarly revisit the methodologies and findings of studies used to determine empirically supported psychotherapies. Although critical of the current state of knowledge, these authors provide practical and well-supported recommendations for future research. Westen et al. (2004) has already sparked ample debate (Ablon & Marci, 2004; Goldfried & Eubanks-Carter, 2004; Haaga, 2004b; Westen, Novotony, & Thompson-Brenner, 2004; Crits-Cristoph, Hollon, & Wilson, 2005; Weisz, Weersing, & Henggeler, 2005; Westen, Novotny, & Thompson-Brenner, 2005). Both Westen et al. (2004) and Doss (2004) promote growth in the field by thoroughly examining and constructively criticizing the existing treatment literature, outlining empirically-founded recommendations, and stirring progressive debate.

Doss (2004): Advocating Constructive Change

Although Doss (2004) argued that the field has not yet agreed about how to study change in psychotherapy, he applauded the field for at least its surge of interest in the mechanisms of therapeutic change. This paper is an exemplary contribution in that it first reviews and clarifies “change” studies and the components that constitute the framework of these studies. It then addresses where improvements need to be made and finally concludes with research strategies based on the current database’s shortcomings and strengths.
By reviewing the definition of “change” studies and highlighting the distinction between change mechanisms, change processes (including both client and therapy change processes) and outcomes, Doss attempts to unite the “outcome” and “process” schools of thought (Haaga, 2004a). His laudable goal to unite rather than divide the field only helps advance current psychotherapy research with greater clarity and precision. He describes “change” studies as research that focuses on the relationship between the active components of a treatment and the treatment’s outcome. Doss characterizes change processes as the variables during treatment that create improvements in the client’s characteristics or skills (e.g., specific treatment intervention (therapy change process), completion of treatment homework (client change process)). Change mechanisms are depicted as clients’ “intermediate changes” that potentially might improve the final therapeutic outcome (e.g., frequency of negative self-thoughts decreases, social interactions increase). The ultimate therapeutic outcome is the final objective of the practitioner (e.g., to decrease overall symptoms of depression, anxiety).
As noted earlier, one of Doss’ primary concerns of the current literature not distinguishing between these components is the polarization of the field between the “process” and “outcome” camps. This divide has led to different interpretations of psychotherapy findings, which has only made our current understanding of mechanisms of change more challenging (Doss, 2004). In order to reduce this gap, Doss suggests that psychotherapy research should focus on the relationship between change processes and change mechanisms rather than change processes and outcome. This narrower scope allows researchers to directly tie specific therapeutic interventions (processes) to immediate client improvements (mechanisms) rather than infer that these interventions indirectly cause long-term change in the client. By identifying key change mechanisms and understanding their relationship to change processes, process research will only thrive within this richer context. The findings of change mechanism studies will provide researchers with a stronger foundation to develop more detailed studies that will identify key change processes.
After pinpointing the shortcomings in the field, Doss proposed a methodological framework that integrates both the “process” and “outcome” approaches. He outlined four phases of research considered necessary in understanding change in psychotherapy, with guiding principles from previous research in support of each step. For example, Doss stipulates in Phase 1 that the field should determine both the efficacy of the treatment and “the nature of the change in the outcome variable” (Doss, 2004). He supports the latter of the two by citing results from a review on CBT for depression by Ilardi and Craighead (1994). In their review, they found that almost half of the treatment gains happened before the specific CBT techniques were introduced into the treatment. Although the results indicate that CBT produced improvements in the participants, the nature of the change (in this case, the timing of the improvements) suggested that the nonspecific change processes of the treatment may have been more useful than the CBT techniques.
Although there are infinite possible ways to advance psychotherapy research, Doss’ framework represents a particularly promising approach. It is clear and testable, and builds upon earlier previous research, and holds promise for integrating two polarized camps of researchers.

Westen, Novotny, & Thompson-Brenner (2004):

Questioning the Status Quo

Westen, Novotny, & Thompson-Brenner (2004) also have contributed significantly to the clinical field by publishing a substantial and thorough review of the assumptions and results of studies used to identify empirically supported treatments (ESTs). This paper, similarly to Doss’ review, critiques the current research methods by questioning the assumptions and the results of the psychotherapy database and then by offering alternative strategies to broaden the clinical utility of psychotherapy research. The debate it has sparked in the field is indicative of this paper’s necessity and timeliness.
Westen et al. (2004) begin by arguing that the randomized control trial methodologies that are used to establish ESTs are based on several assumptions that are not well validated. Similar to Levant (2004), they contest the assumption that research should be based on the notion that patients typically have a single identifiable DSM diagnosis. Unlike Levant, however, these authors simply do not indicate that this assumption is never valid but rather provide the conditions that make this assumption generally invalid. For example, Westen et al. argues that the DSM-defined categories are themselves based on committee consensus rather than empirical methods (e.g., factor or latent class analysis). It is unclear if many of these diagnoses are distinct disorders or perhaps particular ranges on a continuum. It is therefore a valid argument that there may be flaws in our current psychotherapy literature by using DSM-IV categories in our research that are not yet empirically-defined. These authors also point out that the findings based on studies using single DSM diagnoses may not be applicable to the general population. To support this notion, they cite Howard et. al (1996) and Messer (2001) which both indicate that one third to one half of those patients who seek mental health treatment do not fit into one DSM diagnostic category since their problems span a variety of disorders or do not meet the threshold of any particular disorder. Westen et al. continue by critiquing other RCT methodology assumptions in a similar pattern. First they highlight the assumption. They next provide examples of the problems the assumption causes and finally conclude by citing examples in the literature.
Although Westen et al. outline arguments similar to Levant’s, they move beyond criticism by concluding with constructive ways to improve the utility of RCTs in psychotherapy research. First they focus on common problems and potential solutions in reporting clinical studies and their results. For example, they argue against the frequent use of wait-list comparison groups in the RCT literature since it has been shown that any 10 to 20 session intervention is more effective than nothing (Luborsky, McLellan, Diguer, Woody, & Seligman, 1997). Very often these comparisons are used to advocate the superiority of experimental treatments. These authors suggest that researchers should exercise greater caution when describing these control conditions and when reporting their results. Investigators tend to draw very specific conclusions that could be misleading based on these unequal comparisons.
Beyond simply better reporting, Westen et al. also argue that alternative methods are more useful than RCTs under certain circumstances. These authors indicate that RCTs are limited when it boils down to examining long-term outcome in polysymptomatic patients or patients who have treatment-resistant symptoms. Based on the shortcoming of previous research, these authors advocate extending the scientific method to the clinician and using correlational analyses to determine interventions that result in positive short and long-term outcome. Once effective interventions have been determined, these then can be moved into the laboratory for further analysis. Their argument is that effectiveness studies should not be secondary to efficacy studies. These community studies that help quantify data from clinical practice will in turn generate hypotheses for subsequent controlled studies.
By questioning forcefully the status quo of RCT methodology, Westen et al. (2004) provoked numerous commentaries. For example, Ablon and Marci (2004) similarly argued against limiting our focus to manualized therapies and supported the use of naturalistic settings as a laboratory to study change processes in psychotherapy. Goldfried and Eubanks-Carter (2004) also concur with Westen et al. that a shift towards therapy process in the database is needed. The gap between research and clinical practice as well as suggestions to reduce this gap are further highlighted. On the other hand, Haaga (2004b), Crits-Christoph, Wilson, and Hollon (2004), Weisz, Weersing, and Henggeler (2004), although appreciative of the thoughtful review, note their skepticism in Westen et al.’s recommendations and continue to advocate RCT methodology. This article made the field stop to think about current practices (both clinical and research) and overall spurred progressive thought and change.

Doss (2004) and Westen, et al. (2004): Exemplars But Not Perfect

Both papers contribute much to the field and do so in a constructive manner; however, they both have limitations that should not be overlooked. For example, focusing on change mechanisms only after establishing the efficacy of a treatment would be difficult to implement. How much efficacy data would be sufficient to warrant further change mechanism and processes research? Would replicated evidence of superiority to bona fide alternatives be required or would one study showing that an active treatment is better than “nothing” suffice? Depending on the answer, there could be a very long or a quite exclusive list of treatments to be further examined. Additionally, Doss (2004) focuses on streamlining treatments, which is laudable and could be helpful for both treatment dissemination and training. However, streamlining current treatments by identifying their most effective elements may not suffice to achieve substantial improvements in efficacy (Haaga, 2004a).
Like Doss (2004), Westen et al. (2004) made numerous recommendations that might not prove fruitful in all cases, as elaborated in the published commentaries on their article. Whereas we do not necessarily agree with all the arguments presented by Westen et al. (2004) or by Doss (2004), we singled out these articles for praise because their thought-provoking analyses and constructive criticism appear likely to inspire further advances in psychotherapy research.

Something that undoubtedly helps to have good mental health is also working on physical health, for this you can count on nutrisystem reviews. In this service you will find the best trainers and physical trainers that will take you to the goals you want. Get the body of your dreams.

REFERENCES

• Ablon, J. S., & Marci, C. (2004). Psychotherapy process: The missing link: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 130, 664-668.
• Bruce, T. J. & Sanderson, W. C. (2005). Evidence-based psychosocial practices: Past, present, and future. In C. E. Stout & R. A. Hayes (Eds.), Evidence-based practice: Methods, models, and tools for mental health professionals (pp. 220-243). New York: John Wiley & Sons, Inc.
• Chambless, D. L., Baker, M. J., Baucom, D. H., Beutler, L. E., Calhoun, K. S., Crits-Christoph, P., Daiuto, A., DeRubeis, R., Detweiler, J., Haaga, D. A. F., Johnson, S. B., McCurry, S., Mueser, K. T., Pope, K. S., Sanderson, W. C., Shoham, V., Stickle, T., Williams, D. A., & Woody, S. R. (1998). Update on empirically validated therapies, II. The Clinical Psychologist, 51, 3-16.
• Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7-18.
• Christensen, A., & Jacobson, N. S. (1994). Who (or what) can do psychotherapy: The status and challenges of nonprofessional therapies. Psychological Science, 5, 8-14.
• Crits-Christoph, P., Wilson, G. T., & Hollon, S. D. (2005). Empirically supported psychotherapies: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 131, 412-417.
• Doss, B. D. (2004). Changing the way we study change in psychotherapy. Clinical Psychology: Science and Practice, 7, 368-386.
• Goldfried, M. R., & Eubanks-Carter, C. (2004). On the need for a new psychotherapy research paradigm: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 130, 669-673.
• Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30.
• Haaga, D. A. F. (2004a). Being systematic and maybe even scientific about planning psychotherapy research. Clinical Psychology: Science and Practice, 7, 387-390.
• Haaga, D. A. F. (2004b). A healthy dose of criticism for randomized trials: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 130, 674-676.
• Haaga, D. A. F., & Stiles, W. B. (2000). Randomized clinical trials in psychotherapy research: Methodology, design, and evaluation. In C. R. Snyder & R. E. Ingram (Eds.), Handbook of psychological change: Psychotherapy processes and practices for the 21st century (pp. 14-39). New York: Wiley.
• Hayes, S., C. (2004). Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behavior Therapy, 35, 639-665.
• Howard, K. I., Cornilee, T. A., Lyons, J. S., Vessey, J. T., Lueger, R. J., & Saunders, S. M. (1996). Patterns of mental health utilization. Archives of General Psychiatry, 53, 696-703.
• Ilardi, S. S., & Craighead, W. E. (1994). The role of nonspecific factors in cognitive-behavioral therapy for depression. Clinical Psychology: Science and Practice, 1, 138-156.
• Levant, R. F. (2004). The empirically validated treatments movement: A practitioner/educator perspective. Clinical Psychology: Science and Practice, 11, 219-224.
• Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., et al. (1999). The researcher’s own therapy allegiances: A “wild card” in comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6, 95-106.
• Luborsky, L., McLellan, A. T., Diguer, L., Woody, G., & Seligman, D. A. (1997). The psychotherapist matters: Comparison of outcomes across twenty-two therapists and seven patient samples. Clinical Psychology: Science and Practice, 4, 53-65.
• Messer, S. (2001). Empirically supported treatments: What’s a non-behaviorist to do? In B. D. Slife, R. N. Williams, D. Barlow (Eds.) Critical issues in psychotherapy: Translating new ideas into practice (pp. 3-19). Thousand Oaks, CA: Sage.
• Weisz, J. R., Weersing, V. R., & Henggler, S. W. (2004). Jousting with straw men: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 131, 418-426.
• Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The empirical status of empirically-supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130, 631-663.
• Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The next generation of psychotherapy research: Reply to Ablon and Marci (2004), Goldfried and Eubanks-Carter (2004), and Haaga (2004). Psychological Bulletin, 130, 677-683.
• Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2005). EBP ≠ EST: Reply to Crits-Christoph et al. (2005) and Weisz et al. (2005). Psychological Bulletin, 131, 427-433.