Why Numbers Lie

So often people are asked to give feedback on a given project or piece of work, or perhaps a service or product or experience they had. And so often you get the “On a scale from 1 to 10…” measurement.

And this is just a lie.  No, it’s not a lie in the sense of actively deceiving someone, more a lie in that a number doesn’t reveal the truth behind the comment.

Numbers & Feelings Don’t Add Up

I am compiling a list of responses from a feedback loop we have built-in for our documentation projects at work. We use a 1 to 5 feedback scoring system and we allow the respondents (it’s all anonymous) to voice any concerns or ways they think we can improve our service. Inevitably the feedback we do get does not always match with the scores. We get 5 out of 5 on a bunch of criteria, yet the feedback people write in indicates otherwise. So why the discrepancy? Because people will click off numbers and don’t want to think about too deeply. But as soon as they can air their feelings or ideas, they do, and it’s much more insightful and can lead to really actionable feedback.

In short, numbers rarely convey feelings and often people don’t say what they mean.

So Why Measure Using Numbers?

Well, the short version is, it’s easily measured and evaluated. Sixty percent of respondents liked ‘X’, or the average satisfaction score was 4 out of 5. Great, but there is no nitty-gritty detail in that. The other reason is that the world is ruled by numbers guys. When you go in front of a manager who is giving you 30 minutes of his/her time, numbers make a bigger (and simpler) impression than do words. And while both are always open to interpretation, numbers “appear” to tell the truth. And we know that isn’t true. Also, it’s easy to grasp a number and not so much a feeling or a concept. Especially not in the IT industry.

What’s the Answer?

Look, you’re going to have to have numbers to justify your arguments, that is the way business works. But if you get a chance to either speak to your potential respondents in an face-to-face or telephone setting, or you can get a blog discussion going, you will get someone’s real feelings and ideas and that is the closest thing to the truth you’ll get. It will lead you down the path of taking feedback and incorporating it into whatever you do to make it better.


Web 2.0 and Feedback – It’s a Delicate Thing

I would be remiss if I didn’t blog about feedback in the Web 2.0 world. It has become utterly integral to the whole feedback loop from a company and personal perspective. If a company doesn’t have a blog or a wiki, it’s losing out on valuable feedback loops, whether it be internal or external feedback. But it’s a delicate balancing act to use these tools. They harbor huge promise and potential, but they  can also spin out of control if they aren’t used correctly. (I won’t go into Twitter just yet — it’s a different animal, even though it’s related to blogs in many respects.)

The two feedback streams I want to talk about briefly are blogs and wikis.


Everyone has one, everyone uses them, it is the place to spew ideas, feeling, concerns, have discussions with the rest of the world (literally). But how can this tool, this medium be used effectively for customer feedback? In short, if you control it, steer it and maintain it, you can garner a huge amount of feedback over a period of time, which can give you a nice snapshot.

If you deliver a service or a product, a blog can be a good way to get a discussion going. GM’s Fastlane blog is a great example where the higher-ups at the company get in on the discussions. And more importantly, start discussions. It can get you in trouble — just ask Bob Lutz and his comments on global warming — but it’s a reminder that controlling the discussion is critical to getting good feedback. Since almost anyone can post a comment, you are open to harsh criticism — and passionate defenders if you take of customers properly.

If you follow a blog discussion, analyze it, and keep it alive, you can mine it for useful feedback and use that to improve what you’re offering.


Wikis are a different animal, and I am not just talking about Wikipedia. I am talking about chipping in, collaborating and possibly making something greater by using the wisdom of the many. Although a wiki is a collaboration tool, it can also be used to garner feedback, and in most business cases, internally. You can post ideas and documents for reviews and get feedback relatively easily. It’s more one-way than a blog, as permissions and editing rights are required here. But it’s a good tool to do virtual feedback rounds with internal experts. Just remember, you can use a wiki for more than just collaboration, but maintain control of who can access it and edit it.

What’s it all mean?

Web 2.0 tools are critical to gather legitimate, actionable feedback, but you have to be careful about who has control of the conversations and how you direct the flow of info.

Getting Feedback One Reader at a Time

I might as well go into my current client’s office in a space suite because it’s sort of like working in a vacuum. I have lots of stuff — like pages and pages — but at this point in the project really very few comments. In order to make the project a success I need the experts to provide feedback. I can only create so much without getting some kind of validation.

I tried the group thing but that was KOD’d by my project lead (see https://getcustomerfeedback.wordpress.com/2009/10/27/no-control-ko/ ). What has worked has been a rotating series of one-on-one meetings. It has a number of advantages:

  • It helps build ownership — everyone that helps out begins to really understand what the project is trying to accomplish
  • I am able to reduce the time I need to get comments as I can target a specific resource for each topic I need to cover and can point their feedback to the specific points I want to cover
  • The conversations seem to travel all over the place so they help answer questions I didn’t even know to ask

How do you make one-on-one meetings really work for you?

  • You must always have something they can comment on. You can’t just have a headline and ask them for the details. You must have at least some details in place already — even if they are 100% wrong. 
  • As you get them to talk, you have to remember that these are the experts and their comments might push you to a new spaces — one small comment might even mean massive changes. But again, these are the experts.
  • Bounce the changes back to them. You have a built-in reason to give them something else to review so take advantage of that.
  • Use the new relationship. If you find you have more questions based on what they said, go ask. As you have the relationship now, they will (typically) give you the space for more time for those last couple of things.

Can’t get a bunch of people to give you a little feedback each? Don’t be afraid to get your feedback one reader at a time.

The Small-Big-Small Approach


One of the things I have learned about setting up a feedback process is that you need a clear methodology before you start a feedback campaign. I have been participating in and leading a work group for a little while now, and have determined that a small-big-small approach seems to work best, but I think it would be applicable to almost any size company where you are trying to get feedback.

So let’s supposed you are tasked with leading a feedback campaign and you don’t know where to start. Make sure you can do the following things: 1) Get commitment from your superiors on the value this brings and so you can promote it, 2) Run the campaign multiple times, not just once,  and 3) go for a small-big-small approach (assuming you got the approval for #2). So what is this approach? Read on.


Assuming you have your target audience for the campaign figured out and have the tools you need to do it (I know, I am assuming a lot), try running  a small feedback campaign, targeting let’s say no more than 50 – 60 people who you know can give you good feedback. By limiting it to this size, you can keep a better overview of the whole project. Also you will probably need fewer people to support you when you start to evaluate the results (if you can, have a small but dedicated team to help you get through this). Granted, you may not get the widest sample information, but it serves as a very good baseline.


After you have completed your first feedback campaign, and you have wildly impressed your superiors with your brilliant deductions from the feedback you received, propose expanding the scope of the next feedback campaign to double the size. That way you can get a larger number of results and compare them to the first campaign. Yes, a bigger campaign means more coordination, more time spent evaluating the responses and just plain more headaches as you have to keep track of all of it. But bigger numbers do count for something. By casting a wider net, you sometimes catch some good fish. Tip: Also use the first campaign’s questions to drive the new questions of the second campaign.


When the big campaign is over and you have evaluated the results, pick out the respondents who gave you the best feedback and ask them to join a blue ribbon feedback panel. Keep it small, under 30 people if you can. this keeps it very manageable and easier to dissect and evaluate. If they have the time, and are motivated, they can become part of something that really improves your product/service. But the questions you pose in a smaller, more focused feedback campaign will have to be very well thought out and more nitty-gritty.

It’s a ton of work to get feedback, but this approach gives you a long-term view and good method of getting the feedback you need.

If You Lose Control, It’s the KOD

So I am in a meeting to talk about the stack of drafts I had just passed around the table in a medium-sized binder clip. There were three classes of drafts in the stack:

  • Comment on these,
  • Just look at these, and
  • I wanted to let you know that I know we need to publish stuff on these topics  but the drafts you have here are not very close yet.

Before I could set the stage and explain on which of the drafts I wanted feedback, the project manager announced “Please get your feedback back to Dan by Friday.”

While I appreciated the support I knew this was the KOD (kiss of death). Small aside, my wife and I used this term (KOD) when my daughter would announce at dinner that she “loved these <add the name of whatever we just ate>.” We knew she would never eat that specific food ever again. So after a “loved these” announcement my wife would turn to me and say “KOD.” This was the same situation.

Even after I said that I was just looking for comments on the first group of drafts, all the team heard was “READ ALL OF THIS STUFF AND GET COMMENTS BACK RIGHT NOW!”

Needless to say, I received no comments and I have to go to everyone and specifically ask for feedback.

The takeaway. Be cautious about what you ask for. Be very clear on what you want you want your reviewers to do. And, get project managers to give you enough space to run your own meetings or else it’s KOD time for you, too.

Get Feedback Before The Release

I am helping out with a customer engagement project where we are trying to get a feedback loop built in to the testing validation process. The idea is to get feedback from the experts on our documentation before it goes out on the market. A novel idea, in many ways.

The whole effort was a result of just asking questions internally at the company as to when we need feedback and what that feedback should include. It isn’t an effort to get a focus group. It’s using our internal experts and speaking to them face-to-face about possible issues.

It turns out the feedback loop internally is lacking. Our area hasn’t even been considered as part of the feedback chain and never was. But it wasn’t due to intentional neglect. As with so many companies, people just don’t think of the whole feedback loop and when it begins.

To make a long story short (this is a blog after all), I did some hunting around, contacted everyone I know and finally found the right person. We talked, I explained my needs and situation, and we agreed on how the documentation can be tested to get feedback from the people who will use it the most. And all this before the product goes out to the customer. Not after it’s out on the market. Problems solved by us, not the customer.

Why Take the Time?

I am taking a series of on-line courses from Social Media Marketing University on using social media in the business world. The courses have been jam packed and really interesting. The extended session on Facebook was last week, It was given by Jennifer Shaheen. And she is one of the best virtual presenters I have heard. So I sent a mail with this unsolicited, positive feedback to Jennifer and to John Souza from SMMU.

SMMU has asked for my feedback after every session — a nice, well conceived survey fronted by the note: “Please take a moment and fill out these quick surveys so we can better serve you as a client.”

It got me thinking. How will my completing the survey help them serve me as a client? The survey did not ask what I would like to see in the future. Or how, in fact SMMU can serve me better. I would rather have them offer me a coupon a free coffee. Or extend my support a month for every survey I fill out. Or offer me a reduced rate off for my next order. Or offer a bonus for the presenter for every 5 star rating.

Their request did not make it personal enough for me to respond. While I was fine with sending out my feedback on my own, I didn’t feel compelled to take part in the company’s survey. My mail touched on just about every category they had in the survey — so it wasn’t the content. It just did not have a strong enough pitch to have me take the time to fill it out.

When you ask for feedback, make it personal and important to them (your customer) not to you (the organization). You will get people to respond, but the more personal you can make it the better the response.