In Episode 12 of Be Well, Practice Well, host Michael McGee, MD, welcomes guest expert Scott D. Miller, PhD, to discuss feedback-informed treatment (FIT) in clinical practice. Dr. Miller founded the International Center for Clinical Excellence, an international consortium of clinicians, researchers, and educators dedicated to promoting excellence in behavioral health services. His research, books, workshops, speaking events, and rating scales on patient sessions and outcomes continue to make significant practical contributions toward improving and guiding clinical excellence across medical disciplines.
Listen to the full conversation below. A slightly edited transcript follows.
Feedback-Informed Treatment: ORS and SRS
Dr. McGee: Dr. Miller, In your work, you talk about four components of cultivating excellence in any field, including deliberate practice, working right at the edge of your competence, relentlessly seeking feedback through coaching and supervision, and having a system of progressive refinement of your skills. Could you talk about how that all fits into feedback-informed treatment as you see it and why is it so important?
Dr. Miller: Feedback-informed treatment began about 30 years ago, with many influences dating back to Ken Howard and his pioneering research in the mid-90s, and was on examining how could we improve the results of care. If you look back at the data about the effectiveness of psychological intervention, it's remained actually flat since the 1970s. That's both for adult clients and children. And so that's been a big puzzle of researchers. We've had an explosion in the number of treatment methods, an expanding number of diagnostic categories, and no improvement in treatment outcomes.
In addition, a study that we did back in 2016, which was replicated during the pandemic by a group of researchers in Germany, showed that clinicians do not, despite what they believe, improve in their effectiveness over the course of their career. If anything, the majority of us steadily decline.
So how could we circumvent that? What was the work around really? And it turns out it's not going to another workshop. It's not hours of supervision, a license, more professional education. What it really is, is getting feedback abouthow we're working with our individual clients. And that meant for many of these researchers, which started out being called patient-focused research, of asking our clients, measuring whether they were connected to us, since that connection is one of the best predictors of treatment outcome, and then whether or not we were actually helping. And more critically, it's not just about measurement, but about actually doing something with those clients when the data indicate that we're not helping them.
Around the year 2000, we pulled together two very simple measures (the Outcome Rating Scale, ORS, and Session Rating Scale, SRS) that measured both progress and the relationship. Those two measures in combination with the knowledge that's been accrued over the past 20 years of use is now called feedback-informed treatment. It involvesmeasurement and attending to the results, that is discussing them with our clients.
Dr. McGee: I've just started using the ORS and the SRS, and I find them brilliant in their simplicity. Could you describe them briefly and how they're used in a session?
Dr. Miller: So the ORS is, as the name implies, designed to assess progress in terms of the client's well-being or functioning in three different areas, their individual well-being, their relational well-being, and their social well-being. And it was based on a much longer tool called the Outcome Questionnaire 45 (Q45) that was developed by a mentor of mine, Dr. Michael Lambert, while I was a graduate student.
We were finding that the OQ was fantastic, but the population of people we were working with, many of them had challenges in terms of literacy and comprehension. So, we needed something simple. The SRS was developed by a mentor of mine, supervisor Dr. Lynn Johnson. And it's designed to assess the quality from the client's perspective of the connection with us, the helping professional. And it looks at whether or not the client feels understood, whether they feel like we're working on what they want to work on in a way that makes sense for them, that's consistent with their values and preferences.
And it turns out that the administration of these scales, ORS at the beginning of the session that we're working with, takes about 20 seconds to administer and score. Again, it's not about measurement, it's about the conversation that follows. In order to know what I'm supposed to do at the present visit, I have to know whether what we did at the last one actually made a difference. And then at the end of the session, we're giving the SRS, and taking a moment or two to reflect: are we in alignment together about what we're working on, how I understand you, what approach we might be using, and whether it fits who you are as a person. This all gives me a last opportunity before the client leaves my office to see if there are any adjustments that need to be made.
Outliers and Fear of Feedback
Dr. McGee: Absolutely. And I find them really easy and quick to do. One of the things that you've written about, and Anders Ericsson had written about it as well, is that the outliers, the real masters in any field, seem to have this compulsion to relentlessly seek out feedback. On the other hand, I've heard people talk about being afraid of what feedback might say and what it might mean about them. And I think there's some underlying sort of psychological preparation for engaging in feedback-informed treatment. What are your thoughts about that?
Dr. Miller: There are two things. I can tell you that after more than 20 years asking for and receiving feedback, that I still always feel unprepared for what people are going to say. And it can be like a slap upside the head when clients say something that seems so obvious and that I missed it. But I'm also grateful that they would give that kind of feedback to me about the work we're doing to see if there's some way that I can improve that.
Everything in our being is against doing more than is expected at the moment. And I don't know if this is a psychological issue or one of motivation. It's just the way our brains work, that once we achieve a certain level of proficiency, we direct our attention elsewhere because our attention is very limited. Those cognitive resources have to be used wisely. Most of us, in fact, master a task to proficiency and then we move on to something else. The people that we've been studying that excel, whether in psychological therapies, in medicine, in surgery, in sports, those folks don't ‘let it be.’ They instead are pushing at the edge of their performance and they're a bit of an odd breed. They're not the usual folks, you know, there are only so many Olympians among us. The rest of us are much more average.
And so there are two steps in this process. The first is to start getting feedback because the latest data, really fantastic work done by a colleague named Jeff Brown, finds that if we just start asking for feedback, we become more responsive to the client and responsiveness addresses what we call random errors. You know our work is not on purpose, not because we have some flaw in our makeup, but simply because you just miss things. By measuring, we can become more responsive.
Now there's a second step in this process, and that is once you begin to measure, if you're really dedicated, you can begin to identify non-random errors. These are pieces of our practice that we bring to the work on an ongoing basis that limit the outcomes we achieve with our clients. And that's really the focus of deliberate practice, our non-random errors.
Is Supervision Relevant?
Dr. Miller: I want to talk about supervision for a minute. There's virtually no evidence that supervision makes a difference of the kind that you and I were required to do in order to be licensed to practice in this field. It's usually shocking because other research suggests that clinicians really value supervision. This is the work of David Orlinsky and Mikhail Ronnestad – they looked at how therapists develop over the course of their careers, and I think that supervision was the second most valued activity in terms of clinicians' sense of professional development. Again, the problem is we can't find any evidence that it makes a difference. So what does this all mean?
People make arguments to say, ‘Well, you know, you need supervision because that will help you be an ethical practitioner and follow the rules.’ Well, you show me the data and I'll believe it. What deliberate practice requires is a coach. A good model for this is to look to Olympic athletes because most of these elite athletes don't have one overall coach and they don’t pick them from a pool of potential coaches. Instead, championship figure skaters have an equipment coach, a dance coach, a choreography coach, an upper body strength, a lower body strength coach. They have people who specialize in areas that may be in need of improvement. And therapists also need that.
We don't need an overall supervisor. You don't need the best supervisor. You need someone who is an expert in the area in which you evince shortcomings. And that means measurement.
Dr. McGee: Yes, really great advice. And I think the key then is finding these coaches who have the skills. Any suggestions on how to find good coaches to help you with your deficits?
Dr. Miller: I do, and let me tell you about an experience I had as an undergraduate student many moons ago. I was in a class called cognitive psychology and we had to write a paper that was critical of cognitive psychology at the time.
I quickly found that one of the big critics of cognitive theory was B.F. Skinner and I tried to read some of his work but it really wasn't speaking to the topic that I felt like I needed, so I called him on the phone. And surprisingly, he answered. And when he answered, I was so shocked that he did, I hung up the phone. And then I had to call him back and apologize because I was ashamed.
He said, ‘What is it you want?’ And I said, ‘I'm writing this paper…..’ An amazing thing happened. He said, ‘Send me your paper,’ and he marked it up and sent me several pieces. And there began a correspondence that lasted about 6 months. Most importantly, I felt like had I been in need again, I could have called him back. It was a critical learning experience, not only for the specific topic but to pick up the phone. So, my suggestion is to do a bit of literature search and then call. Call the expert, call the person at the top. I find that these experts frequently have a humility that is not typical of their stature in the field.
Barriers to Feedbak
Dr. McGee: Such great advice. Can we talk briefly about the barriers and challenges of feedback-informed treatment that you've encountered?
Dr. Miller: So the biggest barrier is time and if you don't protect your practice time, everything will encroach upon it because deliberate practice is painful and slow and boring. It's sitting and doing….. It's what happens when you're not seeing a client.
What you do in the room with a client is a performance. It's not practice. So, you have to set aside that time and find and protect it. Turn off your computer, turn off notifications and tell your family, don't bug me during this particular time.
Dr. McGee: Let's say that you're practicing empathy….
Dr. Miller: So we just finished a study that is currently under review called the Difficult Conversations in Therapy Project…. there are lots of nuances to empathy. So we first have to figure out where is your particular deficit in empathic practice.
Dr. McGee: Right, Is it simple reflections or complex reflections or over-reflecting or under-reflecting? Is that what you mean?
Dr. Miller: It could be that. It could also be that we, for example, are very good at identifying explicit emotions. We are less capable when it comes to the implied emotion. So, we might see that a client is angry, but we don't comment on or note that they're also quite disappointed. Or it could be the target of the client's emotion. We notice that they aredisappointed, but we don't acknowledge that disappointment is in us and the failures that we had with them in the room in terms of relational responding.
So to deliberately practice empathy, we have set up in this particular study a series of scenarios. And you watch this brief scenario and write out your response. There's then a possibility to give youdetailed nuanced feedback and we feed that information back to you on an ongoing basis while you're watching.
Here's the boring part: watching the exact same video over and over again. And what we're hoping for is that we start to see an uptake of the particular skill that we're trying to teach you in the feedback that we're giving.
Dr. McGee: That will be great technology once it becomes widely available. I'm reminded of Bill Miller's [no relation to Scott D. Miller] work in terms of setting up clinical practice where everybody brings in a 5-minute clip and everybody shares them with a group of clinicians in a safe setting. And I love how everybody can say something positive about it and then one person draws a number out of a bowl and can then say something negative or more constructive. Do you think that that's another way to do deliberate practices by getting together in groups and practicing particular skills?
Dr. Miller: Feedback is a complicated endeavor, and you need feedback from people who possess knowledge and a skill that you don’t. When we look at the data, I don't see evidence that therapist's empathic responsiveness improves over time, and I don't see a correlation, hold on to your chair, between how well the therapist thinks they're responding and the client's experience, I would say it worries me.
Dr. McGee: Yes, that's dangerous, isn't it?
Dr. Miller: It is dangerous because confidence leads us not to question what we're doing and to be at the edge of our performance level. Now I want to I want to make clear that in all this discussion that we're talking about, being at the edge of our performance, that doesn't mean that most of us do bad work. That is not the case. I'm super impressed by how hard clinicians work and work to improve.
Dr. McGee: Yes, we're talking about going from good to great, right? And about 10% of clinicians actually harm their clients. So, instituting something like FIT, where this becomes a cultural norm is crucial.
Any last pearls of wisdom that you would like to share?
Dr. Miller: I have simply done what I felt motivated to do 30 years ago when I went to graduate school. For me, the work that we did made very little sense. I wasn't a natural at this activity. I'm still not a natural. I think when you scratch the surface of most clinicians, they readily say that they're much less certain than they pretend to be or that others think they may be….. Those thoughts sit at the back of your mind. Let's let them free.
Listen to this and other conversations on clinician wellbeing and achieving clinical excellence on your favorite podcast platform.