E. David KLONSKY *
Department of Psychology, University of British Columbia, Canada
Keywords: psychotherapy outcome research, empirically supported treatments, society of clinical psychology.
Coyne and Kok (this issue) argue that the state of psychotherapy research is dire. They further argue that substantial improvements in methodological standards and practices are necessary for psychotherapy research to meet even minimum scientific standards. I can share a particular perspective on this issue. From 2007-2012 I served as Editor of the Society of Clinical Psychology’s (Division 12, APA) list of empirically supported treatments (PsychologicalTreatments.org). This position put me in regular contact with the latest research on psychological treatments of all varieties. And my response to Coyne and Kok is easy to summarize: I agree.
My unqualified agreement with Coyne and Kok may seem surprising given the extensive time and energy I devoted to maintaining the Division 12 list of empirically supported treatments. After all, the website utilized the ‘Chambless criteria’ – a set of specific, objective, and well-known criteria for determining the efficacy of treatments (Chambless & Hollon, 1998) – and the website lists dozens and dozens of treatments that meet these criteria. In brief, to meet the highest standard of “well-established” (termed “Strong” on PsychologicalTreatments.org) a treatment must be supported by at least two independently conducted well-designed studies or by a large series of single case design experiments. Characteristics of a well-designed study include use of a treatment manual, a well-characterized sample, and random assignment to treatment and control conditions. To meet the standard of “probably efficacious” (termed “Modest” on the website) a treatment must be supported by at least one well-designed study or a small series of single case design experiments.
These criteria have substantially impacted the field. Published versions and updates of these criteria have been cited more than 4,000 times according to Google Scholar (e.g., Chambless & Hollon, 1998; Chambless & Ollendick, 2001; Chambless, Sanderson, Shoham, et al., 1996). Arguably, the Division 12 criteria helped bring about the field’s recent emphasis on evidence-based practice (APA, 2006; Kazdin, 2008; Norcross, Beutler, & Levant, 2005). The criteria were ground-breaking, and moved the field forward like few other innovations.
However, it is also true that the landscape of treatment outcome research was very different when the criteria were developed. Two decades ago, when outcome studies were rare, it was indeed plausible to construe two studies supporting a given treatment as representing strong evidence. Fast forward to 2014: there are now thousands of randomized controlled trials, and tremendous advances in methodological sophistication (many of which are emphasized by Coyne and Kok). It is time to thank the original Division 12 criteria for all they have done for the field (and they have done a lot), and move on to more methodologically sound, scientifically valid approaches to generating and quantifying psychotherapy outcome research. The field would do well to heed the recommendations of Coyne and Kok.
Unfortunately, culture change is slow. Psychotherapy studies have been conducted a certain way for two decades and it will be difficult to effect the kind of sea change envisioned by Coyne and Kok. Therefore, I would like to use the rest of this commentary to list initial steps that I believe can facilitate the most change in the shortest amount of time.
Step 1. Division 12 should publish new criteria. I noted earlier the substantial influence and citation count of the original Division 12 criteria. In my experience as Editor of the Division 12 treatments list, many psychologists seemed to reify the criteria, as if they were synonymous with the concept of ‘empirically supported treatment’. Of course there are numerous perspectives and debates regarding how to quantify empirical evidence for psychological treatments (e.g., Borkovec & Costonguay, 1998). In fact, David Tolin, current President of Division 12, is spearheading a committee charged to substantially update the Division’s perspective on how psychotherapy research evidence is conceptualized and quantified (full disclosure: I am a member of this committee). Given the influence of the original Division 12 criteria on scholarship and outcome research, it is likely that a strong update to the criteria published by Division 12 would get the field’s attention, and help fast-track the kinds of changes advocated by Coyne and Kok.
Step 2. Increase sample sizes. Perhaps no single limitation affects the validity and generalizability of psychotherapy outcome research as much as small sample size. As Coyne and Kok note, small sample sizes are the rule rather than the exception in psychotherapy research, and this limitation result in countless false positives that render the literature difficult to interpret. Unlike some other critical recommendations by Coyne and Kok, increasing sample size is a very specific, concrete step that can happen quickly if psychotherapy researchers decide doing so is a priority. An increase in standards for sample size can be encouraged easily if journals update their criteria for what is considered publishable, and simply refuse to publish underpowered outcome trials.
Step 3. Emphasize effect-sizes rather than statistical significance. The original Division 12 criteria rely on statistically significant results to determine if a given treatment is more efficacious than a placebo. However, the better question in any domain of healthcare is not whether a treatment is better than nothing, but how much better. A statistically significant improvement does not at all imply that a patient has been cured or experienced a meaningful improvement in symptoms or quality of life. A focus on effect-sizes, including measures of clinically significant change (e.g., for examples see Jacobson et al., 1999; also see Laupacis, Sackett, & Roberts, 1988), would: a) help the field move away from the categorical, ‘box-score’ approach utilized by the original Division 12 criteria, b) encourage more precise and sophisticated comparisons of the relative efficacy of various treatments, and c) facilitate meta-analyses of therapy outcome studies. Like Step 2, this step would be relatively easy if prioritized. The data necessary to calculate effect-sizes and indices of clinical significance are already collected in typical treatment outcome studies.
In conclusion, there is little question that the field of psychotherapy outcome research needs a major overhaul, and would benefit substantially by taking the recommendations of Coyne and Kok. Coyne and Kok provide a thoughtful and important vision. We must be equally thoughtful in determining how to make this vision a reality.
REFERENCES
APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271-285.
Borkovec, T.D. & Costonguay, L.G. (1998). What is the scientific meaning of empirically supported therapy? Journal of Consulting and Clinical Psychology, 66, 136-142.
Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7-18.
Chambless, D. L., & Ollendick, T. H. (2001). Empirically supported psychological interventions: controversies and evidence. Annual Review of Psychology, 52, 685-716.
Chambless, D. L., Sanderson, W. C., Shoham, V., Bennett-Johnson, S., Pope, K. S., Crits- Christoph, P., . . . McCurry, S. (1996). An update on empirically validated therapies. The Clinical Psychologist, 49, 5-18.
Jacobson, N.S., Roberts, L.J., Berns, S.B., & McGlinchey, J.B. (1999). Methods for defining and determining clinical significance of treatment effects: Description, application, and alternatives. Journal of Consulting and Clinical Psychology, 67(3), 300-307.
Kazdin, A. E. (2008). Evidence-based treatment and practice: new opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63(3), 146-159.
Laupacis, A., Sackett, D.L., & Roberts, R.S. (1988). An assessment of clinically useful measures of the consequences of treatment. New England Journal of Medicine, 318 (26), 1728–33.
Norcross, J. C., Beutler, L. E., & Levant, R. F. (2006). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. Washington, D.C.: American Psychological Association.
