(originally published in Customer Management Insight by ICMI)

Getting a Real Measure on Satisfaction (Part II)

The impetus for this two-part series is to identify why the customer’s perception of service continues to plummet, even while our internal satisfaction surveys receive more and more attention and seem to indicate that satisfaction levels are high.

In Part I, we addressed some of the more tactical problems with our surveys — how the view of the customer is a bit too narrow, and how we ignore the input that is obvious in customer behavior in favor of what is written in surveys.

Part II looks at the side of satisfaction surveys that is a bit more seedy and a bit more controversial. Here we are going to call out the charlatans who are manipulating the results for their own favor. How does that happen? There are two key places where this takes place:

1. Post survey, where the results are delivered in a way that suggests they are better than they truly are; and
2. 2. Pre-survey, where we stack the deck to make sure we get the results we want.

The Post-Survey spin

Let’s start with the post-survey issues. Now, I’ve never run into any organization that just out and out lies when tabulating the results. That is not to say that it isn’t happening, but my best guess is that the instances of it are rare. Much more common is the “spin” on the analysis, and it comes at you from all sides:

• Comparisons are made to whatever timeframe (last month, last quarter or last year) that presents current results in the most favorable light (e.g., “Our 88 percent satisfaction rate is well below last month’s 91 percent, so we’ll compare it to last year’s average of 87 percent so that we can indicate an increase.”).
• The definition of “satisfied” is adjusted to best fit current circumstances — sometimes we look only at “top box” results, sometimes it is everyone who rates us a 3 or above.
• The results are good no matter what. This happens when there is no goal surrounding the numbers, or nothing to compare to. Under those conditions, 91 percent satisfaction sounds just fine. So does 84 percent. And 87 percent.
• We reach for external reasons to justify low numbers, but never to offset high ones. A financial services company, for instance, might suggest that poor investment results “bled over” to low customer satisfaction scores related to call center service. The same organization will never suggest that high service ratings were exaggerated by unusually high financial returns.
  

You might argue that this sort of manipulation occurs just about everywhere you find data. As British politician Benjamin Disraeli (1804-1881) stated, “There are three kinds of lies: lies, damned lies and statistics.” Fair enough, and you get no argument from me on that point. But we are exploring why the plethora of customer satisfaction surveys being taken today aren’t getting us any nearer to great service, and this type of “spin” cannot be ignored as a reason.

Pre-Survey Pleading

Even more devious, however, is the manipulation that occurs pre-survey. At some point in the past few years, some organizations have taken to pressuring customers into providing the highest ratings. By now, you are all familiar with the practice, as I’m sure you’ve been a victim at some point in time. The survey is offered up, but along with it comes pleading and cajoling (often written on what I call a “begging card”) that all the ratings be perfect. The implication is pretty clear that the organization wants only to see perfect scores. The masters at these surveys make you feel guilty about even considering offering a rating that’s less than perfection, suggesting instead that you should contact them to correct the problem rather than filling out the form.

Plenty of people have documented their experiences around this (for example, one of my favorite writers/speakers, George Walther, posted a related entry on his blog archive at www.georgewalther.blogspot.com). One of the many that I’ve incurred recently happened at a rather expensive hotel outside of a major metropolitan city. The nightly room rate was considerable, for which I got a pretty average room, fewer than 20 stations on the TV (and none of my favorites), uninspired food, a missed wake-up call… and a begging card. And before anyone suggests that the card is at least an attempt at providing good service, let me point out that, if my satisfaction was the chief concern, the begging card would never mention a satisfaction survey.

When did it become my job, as a customer, to make sure that I can rate a company all 10s? It’s not that I mind providing the constructive feedback (in truth, I rather get a kick out of it), but I’m much less motivated to do so if I feel that the main reason is to bump up their standing in the corporation. I’d prefer that my feedback be used to improve service for everyone coming after me. In momentary lapses of sanity, I sometimes even dream that the feedback might be appreciated by management and cause them to rethink their operation from the perspective of the customer. For crying out loud, at $279 a night, is The Golf Channel or the Food Network too much to ask for?!

Getting the real results

So what can we do to get a real rating on customer satisfaction? We can start with expanding our view of the customer and integrating customer behavior into the equation, as pointed out in Part I of this series. After that, we need to make sure that we’ve built integrity into the process. If you use some form of begging cards, burn them. Then determine exactly how you are going to report on customer satisfaction and stick with it, thereby removing the “spin” temptation. Accurate satisfaction ratings are missing from too many of our organizations, and we will never close the gap between real and reported satisfaction until we true up the numbers.

Of course, we can always attack it from the other angle. We’re all customer service professionals, but we are also all consumers. We get bombarded with the begging cards all the time. Let’s start writing a little note on them indicating that whenever we get a begging card we automatically rate the organization as low as possible on the survey. Maybe that will shift the responsibility for ensuring satisfaction back to where it belongs.

Jay’s Formula for Data Integrity The core of the satisfaction survey problem is that we’ve taken what could be a great metric for measuring and improving an operation, and turned it into a metric we use to rate an organization (or a person within that organization). Whenever we do that, the integrity of the metric falls drastically, as quantified by what I call “Jay’s formula for data integrity.” It goes like this: dir = 100/(Rm + (Ri x Rp)) Where: dir = data integrity rating Rm = Rater manipulation (zero to 50 scale) Ri = Ratee impact (how the ratings will affect those being rated, on a 1-to-5 scale) Rp = Ratee power level (how the ratee can retaliate over the rater, on a 1-to-10 scale) In the hotel room example, the ratee is the General Manager of the hotel, and as the guest, I’m the rater. I’m guessing that the impact of my ratings matter greatly at this hotel (let’s give that a 5), as evidenced by the begging card. The hotel has little power over me (just a 1 rating), but they are certainly at least trying to manipulate me (I'll give that a 40 on our 50-point manipulation scale). All in all, it adds up to a paltry dir of 2.2 (100 divided by 45) for their satisfaction scores. Take away the begging card, and Rm goes down to zero, bringing the dir up to 20 (100 divided by 5). Note: Want a really low dir rating in a contact center? Have your supervisors do all the monitoring for their teams and provide them with a financial incentive to report a high-quality rating for the team. You’ve effectively made the supervisor the ratee and the rater, and you’ll get the lowest possible dir of 1 (100 divided by 100). Comments or questions on the formula? Email me at jaym@serviceagility.com.

- Jay Minnucci
President
Service Agility
(215) 679-5250
jaym@serviceagility.com