Journal of Online Research

Interview: Jack Honomichl on the State of Online Research

Wed, 24 Oct 2007 22:43:18 GMT

PDF version available here.

Jack Honomichl is considered one of the foremost authorities on the research industry, and for good reason. The Market Research Council, an organization founded in 1927 to foster growth of the advertising/marketing/public opinion research industry, credited Jack Honomichl as the person who “defined the market research industry” at his June 21, 2002 Hall of Fame induction ceremony at the Yale Club in New York City. Mr. Honomichl shares Hall of Fame status with other inductees who have also been recognized for their extraordinary contributions to the industry, including renowned research authorities: Arthur C. Nielsen Sr., George H. Gallup Sr., David Ogilvy, Marion Harper, Daniel Yankelovich, Daniel Starch, Ernest Dichter, Alfred Politz, and Elmo Roper.

Since 1969 Mr. Honomichl has written about the market research industry, as publisher of the first research trade journal, The Analyst, and Advertising Age research industry columnist for 18 years, and American Marketing Association’s (AMA) Marketing News research columnist for five years. He authored an industry textbook, Honomichl on Marketing Research, has written nearly 400 articles about the industry in the trade and academic press, and has been a featured speaker at national research industry conventions and trade association conferences, and has guest lectured at several universities.

On October 9, 2007 JOR Managing Editor Steve sat down in conversation with Jack to get his perspective on the state of online market research.

Transcript Provided By:
Transcription Services, Inc
(909) 484-8185
www.tsitranscripts.com

SA: Thank you for joining us today, Jack. I’d like to start off by taking stock of the current state of online research and how it currently fits into the market research mix. Online has been a viable option for fielding research studies for almost a dozen years now. ESOMAR reported that in 2006, out of $24 billion US dollars spent worldwide on market research, $3 billion US was spent on online market research, a 14% increase on the year before. What is your take on these numbers? Do they match what you are seeing? Is there more to the picture of where we are now with online research?

JH: Yes. The data that we prefer to use is that developed by Inside Research where we get the major users to provide us in a confidential basis, their spending via various methodologies including online. We break that down by the United States and Western Europe. Let’s talk US only for a minute. In 2006, online spending was up by 19% and the estimates we have for 2007 are up 15%. That’s pretty healthy, but it shows the start of a topping off if you will. The growth rate was very; very steep in percentage basis in previous years, but now logically it will start to plateau. In Europe we see pretty much the same picture. The one basic difference is we’re talking Euros instead of dollars. Growth in all the Western European countries put together was about 54% in ‘06 and 29% estimated for ‘07. So you can see that it’s following kind of the same pattern. But it boils down to this, that in the US about 36% of all survey work done was via online data collection in ‘06 and the estimate for ‘07 is 39%. Those are the numbers we go by. We have reason to believe they are accurate except of course they are not all inclusive. They don’t represent every last penny that’s spent out there. But as far as the growth rate is concerned I’m sure they are quite accurate. People seem to agree with us on that. There is a difference though – it might be of interest to your readers – there hasn’t been universal acceptance of online for various types of projects. It’s different in Europe than it is in the United States. For instance, the Europeans apparently have a different take on the value of online. In the United States, the leading usage is concept and product testing. In Europe, the leading use is also product and concept testing. But when you come to the next most important thing, in the United States it’s sales tracking and in Europe it’s customer satisfaction measures. So you go on down all the possible applications: A&U studies, advertising brand tracking, opinion polling, sales tracking and so on, there is quite a bit of difference. For instance, in the United States, sales tracking is number two. In Europe it’s way down the list, only about 6% of the total. So they’ve adopted it, but with a significant difference in the application they put it through.

SA: So concept and product testing is still the number one use on both sides of the pond.

JH: Both sides, yes. If you add them together for the world, that’s it by far. And I think the real message for this is that when we talk about the embracing, if you will, the concept, the data collection online, the beauty is inside the beholder. They see it as – their experience has told them it works well for this application, but it is not so good for other applications that sorted itself out over time; you know, trial and error. This is the way it has kind of shook down. In the United States, just to refresh your memory, the listing goes concept product testing, sales tracking and A&U studies, brand test tracking, CSM (which is very big in Europe), copy testing, site evaluation, user profiles, opinion polling, qualitative focus groups and others. But by far, a third of it is just concept and product testing.

SA: You mentioned qualitative such as focus groups. Are you seeing any changes in that? Is there more wide adoption on the qualitative side of the coin as well as the quantitative side of the coin?

JH: It’s interesting because if we can talk tangentially, as you well know, there has been a splurge of interest in recent years with the so-called “proprietary panels” and especially those that are called “community panels.” For those people who aren’t familiar with community panels, a client owns them. They recruit them. They might recruit them from their own customer body and they are really used as sounding boards. They are a two-way conversation link with knowing customers perhaps. These community panels are almost entirely used for what I would call “qualitative type insights” as to how your best and most important customers are reacting to new product concepts or a new advertising campaign or a new product, whatever. So in that sense, the qualitative end of it is getting a very special shot in the arm because of the growing acceptance and usage of these so-called proprietary panels which are owned by the client, and are focused on a very specific population. Quite often that population is comprised of their own customers selected or taken from a sales list or recruited at the point of sale, or something like that. It’s not so much the quality of the sample as it is the very richness or concentration of the sample in the sense that you know these are our heavy customers. Now we just want to tickle them with a new logo, a new this or a new that or a new product concept and see how they bounce. But we want to have two-way communication always through the process. The growing use of online for what I’ll call qualitative inquiries will have a big impact over time.

SA: Are you seeing any numbers around that in terms of percentage growth in that particular…

JH: We set up a panel of people who are – companies that are specializing in the sale and development of these so-called proprietary panels of both community and non-community. That was up by 26% by midyear in ‘07 and its projection should be up 37% by midyear ‘08. And the projection for October is up 37%. Revenue is getting close to $100 million, which is pretty sizable. I liken it very much to the original growth of customer satisfaction measurement which started off with just a few firms, highly specialized, developed a concept and methodologies and then everybody and their brother jumped on the bandwagon and said, “We can do that, too.” It became a big growth sector for the industry. I think these proprietary panels could be the same thing; they show all the signs of growing and developing, just like customer satisfaction measurement did.

SA: That’s very interesting that qualitative is starting to make an impact as well. Speaking of impact, can you assess the impact – favorable or otherwise – that online has had on the market industry in general in terms of benefit to methodology – are we doing more useful work, serving our clients better? And in terms of how online is growing our industry as well.

JH: Well my impression – and I underline the word impression, is that online was embraced the way it was for basically two reasons. One is that most research managers are being pounded by their masters to be fast and quicker; do things overnight and online promised that. The second was cheap, it’s not as expensive. And that was a big lure: fast and cheap. They jumped in because both of those were huge incentives. Then it was only after they got into it, that they started to realize “Well, it can be fast and cheap, but no good at the same time.” Then the sorting out process I described earlier – it’s good, okay for concept, product testing, but it’s no good for say polling, kind of sorting out came. And now we get down to that it’s matured to the point where people can look at it and say, “Well, it’s a fact of life, it’s established. We kind of sorted out the types of studies we’ll be using, but now what do we think about the quality of the data?” It turns out that’s quite a concern. We, being Inside Research, have built a panel of directors of research, very major buyers – 35 people who run the largest research company departments in the US, basically. It’s not exactly true, but that’s the essence of it. We asked them recently about their concern about the quality of online panels. The majority of them said it was one of their special concerns. Over half of them said it was “one of their special concerns” in running their operation and about a third of them said it was “the most pressing concern.” We asked them if they’ve seen improvements in online quality. The answer is “No, not necessarily. It’s staying about the same and, if changing at all, it’s probably deteriorating,” How do end clients face up to this? . Well, some of them – first of all, they do the usual thing; they say it’s the vendor’s problem. “We expect the vendor to solve it.” But the essence of it is that they have been very reactive to the vendors saying, “We are doing our own R&D. We are concerned about respondent quality, representatives and so on. Here is what we are doing; here is how we are trying to improve or even eliminate the bad stuff, the professional respondents and that sort of thing and improve the quality of the sample.” Now the research companies, the vendors, who talk that way and bring R&D projects in and so on, are getting the business. The other guys who aren’t paying much attention to that are just saying we are cheap and fast, might get the business in the first place, but eventually they will lose out to someone who says, “Hey, we’re taking this seriously, this sampling problem, we’re trying to rectify it or at least keep it certain parameters.” These guys say that they very definitely have dropped vendors who didn’t show that willingness to work on sample quality and give more business to those who are in fact doing in-house R&D, trying to find ways to improve on the process. That’s the way the world works. People eventually move their business to where they feel more comfortable. They want to deal with people who are serious about the problems, who really want to do a good job. They gravitate away from companies whose main pitches were all about ‘cheap.’ That should be no surprise. But there is no question that to research directors, this is a real worry. And especially some are after the revelations of how much overlap there was between the online panels – and we have professional respondents and people doing this to make a living practically, getting remunerated and being parts of five or six different panels. That’s been documented by Comscore. And that was a very sobering piece of information and it shook a lot of people up. Of course there have been all sorts of symposiums and seminars and conference programs devoted to the subject in the last few months and into the future on that very subject. So people are worried but as usual the buyer says, “Well it’s the seller’s problem and he’s got to do whatever is necessary to solve it.”

SA: Is there any indication of willingness on the buyer’s side to pay a little more for what they perceive as more quality, or is price pressure still there?

JH: That’s always the question. That was the question back with telephones. It will always be the question. “They want it, but do they want to pay for it or are they willing to pay for it?” I don’t have anything substantive documentation to put towards that. But what I do see – and this is probably true in all the research they do, a handful of end users take it so seriously that they will say assign one person to the subject – let’s say quality of online samples. They actually invest in someone inside and make them a warden or a monitor to work with the research vendors on just that subject. Well, that costs money to take one staff and in effect make them the in-house expert, if you will. Most marketing companies won’t do that. You would hope they would. One of the problems getting an answer to the question you asked, Steve, today is the expanding role of the purchasing department. Those guys bring a whole new dimension to the process. They aren’t nearly as swayed by some of the subtleties that research people would recognize and appreciate. These guys operate on a cost basis as if they were buying a commodity– there is no black and white; everything is just gray to them.

SA: So thinking past the sample issue, which is obviously a bit concern.

JH: That, and professional respondents, yes.

SA: In terms of the actual information that research clients are getting – they are working with a vendor who is paying attention to sample quality and professional respondents…

JH: They tend to gravitate to them over time.

SA: Then for those people, do you get the sense that the end clients are feeling they are getting what they are paying for for online? Are they satisfied with it? Are they getting the information they need to do their jobs?

JH: If they keep buying it, you have to conclude that if not entirely satisfied, they find the deliverable at least tolerable. I think that’s the proof of the pudding. They can say anything they want to, but if they keep buying it at the rate they are going and it keeps continuing to increase the amount of usage, you have to conclude that it’s serving their purpose. Now deep in their hearts, they may find it inferior or flawed. But perhaps it gives them a speed element that makes them heroes to their marketing management; that alone makes it all worthwhile. So I would say it’s serving its purpose. But it does bring up to me a very famous saying. I don’t know if you ever took courses in economics when you were in university, but if you did you probably ran across Gresham’s Law. It said quite simply, “Bad money drives out good.” That’s an economic truth. In that sense, online research could be the bad money that is driving out good research practice. But it does turn right around and get accepted because it has this evil virtue of being so fast and relatively cheap.

SA: But is that to say then that it is inherently inferior because of the medium?

JH: If the management doesn’t appreciate the difference in quality, and doesn’t even want to listen to it, who knows?

SA: So what you are saying is there is a difference in quality.

JH: I don’t want to say that. I guess you’d have to get right down to any particular study done on any particular type of project for any particular client. Generally speaking, we do know that there is a lot of concern among people who really understand good research methodology from an academic point of view. We’re getting further from that standard of excellence. Does that mean that there is no utility in the data? I didn’t say that at all.

SA: Can you address some of those concerns? Obviously the sample is a big one. Are there other concerns besides sample?

JH: I have my own hunches, but I don’t have any numbers to look at to tell you something solid. . I’d rather stick to areas where we’ve made inquiries and have some numbers to back-up what we’re saying. I’m much more comfortable doing that.

SA: Returning to the sample issue, what do you think can be done? If you had to prescribe something to the online side of the industry what would you prescribe on the sample issue?

JH: I would say that if you want to stay in the business and be a major player through time, you should spend money on internal R&D. BASES, in my opinion came to dominate the market modeling business for a simple reason, they spent so much money on R&D and they worked closely with the clients. They told them exactly what they were doing and why they were doing it. They invested the money, which also included hiring some very high-powered academics to go off and sit in the corner and work on these problems. Over time it really paid off because the clients came to believe in what these guys were doing.. Of course, BASES is basically in the business of forecasting the acceptance of new products and there was proof in the pudding. Did BASES predictions work out well? So there was kind of – like an election. There was a result and everybody could say yes, BASES was right. But the biggest thing was that didn’t just happen. BASES made it very clear to their clients that it was an on-going process; you just don’t develop a methodology and say there it is and that’s it, see you around. They kept pumping money in and they continue to this day to pump money in to R&D. Going full circle, Steve, I think if someone wants to continue to be a major player in the area of supply and online survey research, they too, will take that posture and spend the money, bite the bullet, do the internal work, have people who specialize, concentrate on this business of quality and show to their clients what they are learning and how to apply it and why they are doing it this way versus that way – they will prevail. And in the process, the quality of their product will get better and clients will believe it’s a better product. So many research companies tend to come up with say a methodology in the case of customer satisfaction. Then they want to just mass-produce it. “This is what we do; take it or leave it.”

I think especially in the case of online, the field is just hungry for people who say, “We realize this is an ongoing process. We’re up to our eyeballs already in it, now let’s find out how we can improve or get rid of some of the problems or alleviate some concerns” and talk openly with the client about it and share with them. They’ll prosper in the long run I think.

SA: Let’s turn from the present and look to the future. Looking into your crystal ball, where do you see online research in five years or ten years out from now?

JH: When we start talking five or ten years out, it’s a fascinating subject to me because things change, especially in today’s world of technological innovation. For instance, we looked at TV and radio audience measurement. One of the largest expenditures for survey research in the United States is traditional TV and radio audience measurement. Traditionally measurements are based on diaries. Then they went to meters and now they are going to personal meters. They find miniaturization has made it possible for someone to carry a little cartridge about the size of a pack of cigarettes around on their body that will record the signals they are exposed to; whether they are in a bar or in their car or wherever they are. This is just now after years and years of development and again, R&D money and so on, it’s just now starting to be implemented in the field in various countries around the world including the United States. It was a transition project that literally started ten years ago. But science made possible the quality of equipment and the miniaturization of the equipment and eventually the lower cost of the equipment so that you can buy it in large numbers and equip panelists with the little portable gadget. You got to look out and say technology can strike from surprising cases. If I had talked to you ten years ago and said do you think some day your child at home will have a pen pal in China and send a thing called an email back and forth to them, you would have said you are crazy. But it has come to be.

SA: So you see technology really is going to drive where…

JH: What we don’t know in the research business really is what’s going to strike again. None of us does. Don’t tell me you knew – go back ten years ago and say that you will come to work every morning and find 100 so called emails on your computer. You would have laughed. But here it is. We don’t know ten years down the road what other technology – which is maybe already developed, has just gotten to the point where it’s cost effective or they can’t perfect the miniaturization part of it that it hasn’t hit yet. It could come along and change everything just as online changed survey research. So when you talk five, ten years out, you got to always say “What kind of technology is lurking in the wings?” It’s just a matter of time and we get one step closer to some sort of Buck Rogers kind of breakthrough that enables us to gather data in ways we never dreamed of. It’s possible. That’s exactly what’s going on in the world of TV and radio audience measurement. It has finally come. People are going to wearing around a little portable receiver that will record all their signals and enable to say that this is what they listen to during the day, commercials. You embed the commercials with a special signal and you got their commercial exposure and away we go. It’s very conceivable to say someday that that receiver could be the size of say a silver dollar and that millions of people could be wearing them all the time. Why not?

SA: And do you see that methodology will evolve, too?

JH: Methodology will follow the technology is what I’m trying to say. You talk about samples; the samples you learned about when you were in college; statisticians and probabilities and all that nonsense. Go to a brand manager and say there are 100,000 people in this panel – and he’s not going to think panel. “Well, how could 100,000 panel size be wrong?” There are that many people out there doing this. “But Nielson samples 2,000.” “Yes, but this is 100,000.” That’s the way their minds work. If you get that number up to there is 500,000 people wearing this little gadget, we’re apt to believe that more than he is going to believe a smaller “sample” in a minute. I think people in the research business have to face up to Gresham’s Law, again.

SA: At the IIR Market Research Event in San Francisco in 2004, I attended your speech calling for less customer insight, which you described as vague and based loosely on intuition, and for more customer intelligence from the market research industry. In your opinion, has online research contributed to more customer intelligence? If so, do you see it as something that is helping to serve the mission of market research?

JH: To me, this whole insight thing to me is a regression. You are going backwards. If you looked up “insight” in the dictionary, they use the word ‘hunch.’ Now do you really want to make big basic investment decisions – like taking a new product to market or building a new plan or something – based on hunches, like something a woman happened to say in a focus group? Or something she did in her kitchen? I mean, do you really want to base that expenditure on an “insight” or do you want hard data?

Hard data suggests that it could come from a lot of sources in addition to what you and I know as traditional marketing research. I think if marketing research people or the industry has a flaw, its being too much wedded to this procedure we call ‘research.’ There are a lot of other ways to get good information that may be much more valuable and helpful. I’ll start off with database mining. With the advent of some terrific software – now when I get to technology, let’s don’t ever forget software. Some of the software that has been developed has made it possible to dive into huge databases and access data like you’ve never been able to do before. Whole companies have been built on showing people how to take in-house databases and convert them into information sources. If you have the right kind of input, the right kind of data and the ability to access it properly with the right kind of software, you’ve got a data source that can potentially overwhelm anything you could learn through conventional marketing research. That’s where I employ the word “intelligence.” The important thing is to get the right kind of information that can be most helpful. Don’t get too hung up on where it comes from. It doesn’t have to come from a research company in order to be extraordinarily useful and relevant – and maybe much less expensive. The research industry has been provincial; it really could have been a major player in the intelligence industry today if it embraced other ways of doing things instead of focusing just on research techniques. Why shouldn’t researchers be consultants to a company on the development of data and the quality of data?

SA: So research companies could be helping companies leverage all the data they have at their disposal to understand their market.

JH: Well the research people can bring their background and their point-of-view to a database mining project and saying, “Hey, look at what’s inherent in that database or look what we can make inherent in there, and here is how we can capture it.” Now, their ability to do that has been somewhat limited by software, but the software people have closed that gap, with the software that exists today. Guess who comes and does that? Management consultants do it. Research people sit there and watch – aren’t even involved.

SA: So if you had a mantra for the industry…

JH: Think big. Instead they have thought small – and you can see why. Researchers have procedures, products, and knowledge to deliver it.. That’s where researchers make their money, but that’s not where the future is. The future is what the client needs, not what works well for the research company today. I’m afraid too many research companies have looked at what works well for us, what works well internally, what we can do easily and simply and make a lot of money doing at present. Well, the real question is serving the client’s inherent need for marketing intelligence.

I have a little case history which will help explain what I’m talking about. At one time in the automotive industry, Detroit became extraordinarily interested in the pricing of cars as it related to leasing agreements. As more and more of the upper-end cars were released, it wasn’t the sticker price that was important, it was the lease agreement. That was the real price competition. So Detroit salespeople say “Hey, we want to know what our competitor is doing in the lease end of the business.” Well, research people look at that and say “Well, interview a bunch of buyers and try to find what kind of leases they signed.” There was one small research company out in Oregon that said, “Every car that goes off the lot has an insurance policy on it as part of the lease mechanism. We can get from that insurance policy all the terms of the lease and a lot about the buyer: age, sex, location, etc. and let’s just do that and short-circuit the whole system” and supplied the data continuously to Detroit on the terms of lease agreements but were getting it from the lease agreements themselves from the insurance companies. Only two or three insurance companies specialize in insuring cars that are leased. “If we can get them to give us their records, we got it.” Research people will still be sitting out there trying to figure out humungous samples of people to be interviewed to find out about their lease arrangements when you can get the same or better information from the lease agreements from the insurance companies. That’s the kind of thinking I’m talking about. You approach the problem not in what is the best research solution; you approach the problem as what is the best data collection system to solve this particular client’s wants and needs. I hope that makes it clear because that’s what I’m trying to say. Start thinking that way instead of saying “We can talk the client into doing 10,000 interviews a month and keep our phone system busy or something.” Stop thinking that way because it’s going to be the death of you. We’re still performing an extremely valuable function, role – because somebody has to do all that; collect the data, process it and analyze it. You still have to do all that.

SA: At the end of the day, somebody has to make sense of it.

JH: But the sales guys are going to buy it. It’s the customer, the guy that is putting up the money, the sales managers. He says, “Where are you getting this?” You say, “Right off the insurance policy.” And boy he’s going to believe it. Somebody else can say, “Well, we’re interviewing people over the phone.” He’ll say, “That isn’t the same. You can get it off the insurance policy.” He’s going to believe the insurance policy approach. You don’t have to talk sample size or any of that stuff.

SA: That’s all the questions I have for you. I want to thank you for sharing your time and your thoughts on the industry and online research.

JH: Thank you.

Sense and Online Sensibility

Jonathan Moody — Sat, 13 Oct 2007 04:18:30 GMT

PDF version available here.

Abstract:

User-generated content (UGC) and social media are constantly growing in number and influence on consumer decisions. Blogs, forums, usenets, audiovisual sharing media, social networks, podcasts microblogging sites and social news sites all offer the opportunity for people to share their experiences of products, services, brands and organizations. Consequently, these media are largely beyond the control of organizations and several large companies have experienced crises that have spread and grown through social media. This article argues that monitoring and understanding social media is essential for many organizations if they are to formulate a strategy for dealing with adverse and supportive comment. Some organizations are taking this one step further by recognizing that when correctly gathered, analyzed and presented, UGC represents a valuable source of insight into understanding and engaging with consumers.

Author Information:

Jon Moody
Market Engineering

jon.moody@asomo.net
skype: jmoody-asomo

ASOMO Service
www.asomo.net

socialware
Garrick House 26-27 Southampton Street
London WC2E 7RS United Kingdom
tel. (+44) 207 717 8475
www.socialware.eu

Paula buys a bottle of shampoo in the local store in a small Arkansas town where she regularly does her shopping. Later that night she washes her hair with it and wakes up the next day with orange hair. After going to the store where she bought the shampoo to complain to the perplexed store clerk, she takes a few photos of her orange hair and posts them on her blog. Like many personal blogs, Paula’s is little more than an online diary and is read regularly by about 4 people. One of these is her sister Anita who lives in New York City and goes onto a health and beauty forum where she posts the story (photos included) of the orange-haired girl. Within 24 hours, the story has spread widely and a manager of a well-known shampoo brand is on the the phone to a communications agency specialised in crisis management.

This story is fictitious but internet lore abounds with such crises: explosive mixtures of candy and soft drinks, easy to pick bicycle locks, exploding laptop batteries. Google the search terms Nestlé + powdered milk, Coca Cola + Columbia or Nike + Thailand and among the first results that come up are websites that present key public relations issues for these companies. These websites create information currents of public sentiment that spread through the Internet to form part of the global rumor network that is consumer generated media (blogs, forums, message boards etc.)

Hence some companies have woken up to this environment beyond their control and at least try to monitor online sentiment to pick up on potential threats and sometimes even opportunities. A select few have taken it one step further and recognise online sentiment as a source of insight and a channel for engagement with consumers and other stakeholders.

The power of online sentiment

Less well known than the aforementioned crises but nonetheless remarkable are the small brands which leverage online sentiment to build up an impressive business. Last year, a wine label called Stormhoek in South Africa used a blog to launch their brand in the UK with the help of Hugh Macleod. They sent a booklet called “Wine Blogging as Marketing Disruption” to 85 bloggers, accompanied with a bottle of their wine.

These bloggers were alpha influencers key to other customers’ awareness, consideration, preference, and purchase. They advocate, rank, sort, evaluate, and ultimately create marketplace adoption. They have a powerful multiplier effect, spreading the word quickly across a broad network when they find something they want others to know about.

The bloggers received a free bottle wine and in turn, maybe two, three, or fifty of them blogged about it. They also told their friends about it in pubs, bars, dinner parties and the ripple effect created consumer awareness and favorability towards the brand that would have cost a considerable sum to achieve by traditional marketing channels. But what would this small wine company from South Africa have lost had it not worked? They would have written off the approximately $400 cost of implementing the campaign and tried the traditional approach. Welcome to the world of word of mouth (WOM) or word of mouse.

David Godes of Harvard University and others provide a good summary of how myriad definitions of word of mouth have developed over time. Since the traditional view of WOM is too narrow to capture the breath of influences, Godes and his fellow authors prefer the term social interactions (SIs). “This term is intentionally general, capturing as it does, a wide and heterogeneous set of contexts. We define an SI as an action or actions that a) is taken by an individual not actively engaged in selling the product or service and that b) impacts others’ expected utility for that product or service.”

To clarify the SI concept, Godes et al set out three essential facets of any social interaction: channel, content and impact. The channel is the medium through which the influence takes place. This might be face to-face discussion, for example. Important dimensions that differentiate various channels include credibility, number of recipients, and whether or not the channel is mediated by a third party. The content is information that is transmitted over this channel. Important dimensions of content include its objectivity and valence. Impact represents the ultimate effect of others’ actions.

Growing in number and influence

Blogs, forums and other online communities are a channel growing in number and influence and references within them the content. Any figures on the total number of blogs, forums, videos posted on YouTube, photos on Flickr or Facebook members will be out of date by the time you read this Log on to these sites, search on Technoratti or Blogpulse or run an enquiry on the myriad sites that answer these enquiries for up to date figures.

Another question altogether is the impact of these new media channels compared with advertising, media coverage and corporate websites. A degree of cynicism is prudent – these are, after all, personal opinions. However, in a recent Ipsos/Mori poll ((http://www.mori.com/polls/2006/blogging.shtml), customer reviews were considerably more trusted than company statements, e-mails and TV advertisements and about the same as newspaper articles.

Indeed, Godes et al point to a number of factors that have lead to some observers stating that consumers’ judgements of the usefulness of advertising have declined over time and is now deemed to be lower than than WOM. These relate to demand-side factors: products are becoming increasingly more complex and hence consumers’ information needs also become more complex.
On the supply side, technological developments in e-mail, internet-based media and mobile communications have increased the number and forms of decentralized communication. Hence information can be transmitted more quickly to more people increasingly cheaply. However, as Godes and his colleagues point out, this is just potential: the decision on whether to share information still resides with the individual. Furthermore, less compelling information might travel more slowly and be “crowded out” by more exciting developments.

Yet it has become much easier to aggregate interpersonal communication. This is particularly true of online discussion forums and strings on blog discussions. Notice the plural in the graph above: Reviews on a recognised review website. If three out of 20 reviews of a hotel are critical, we might still go ahead and book a room there. The more that negative proportion increases, the less inclined we will be to stay there. According to the Ipsos /Mori poll, 39 million Europeans decided not to buy a particular product after reading negative comments about it on the Internet. There is often a direct relationship between a wave of positive or negative online sentiment and sales figures. Studies of those relationships and even between online sentiment about listed companies and their respective share prices would make for an interesting line of research.

So, with the myriad sources of information at consumers’ fingertips, why are an increasing number of people visiting specialist forums or leading blogs before buying a mobile phone, taking out a mortgage or booking a holiday?

They want to hear what fellow consumers think about a product or service before making the purchase decision. Car buyers can visit the brand website or even read a car magazine review waxing lyrical about car X’s beautiful lines or superb handling. But what about those annoying problems that sometimes emerge a couple of months after taking delivery of the vehicle – doors that jam, uneven steering or scraping windshield wipers? Log onto to a forum on car X and find out about these and other issues from people who already own the automobile. More positively, you might be 85% decided that this is the vehicle for you and seek out positive references to finally convince yourself (and perhaps another decision maker).

Fake and be damned

So isn’t this a golden opportunity for managers to get online and blow their own trumpets?And why not denigrate the competition while you are about it too? After all, myriad forums are available to everyone or you can even set up your own blog. Many travelers have alleged that the influential Tripadvisor portal is riddled with glowing reviews of hotels written by the hotel owners or managers. How do companies offering social media insight services get round this? These reviews are sometimes glaringly obvious because they are usually short – lack of detail is often a sign of lie. However, some clients want them included in the analysis (often tagged as suspect) because they can still influence perception whether they are are genuine customer reviews or not.

Furthermore, social media lore also includes stories of attempts to influence perceptions which have proved to be unethical, disastrous or both. Whole Earth’s CEO John Mackey recent bashing of the financials of Wild Oats (under the moniker “harobed”—his wife’s name spelled backwards on Yahoo) to damage Wild Oats stock price before he purchased it. US retailer Wal Mart received a wave of criticism when it attempted to launch its own social network. The Hub, as it was called, was supposed to be a place where high school kids could socialise and make friends. However, visitors became increasingly suspicious when they read some of the comments that were very positive about the brand. Word spread that these and been written by marketing professionals and the site closed down after four months. McDonald’s was also caught using a flog (fake blog). The blog in question appeared to be a written by a security guard who was so fascinated with a marketing promotion, that he wrote about it in his blog. Another blog call The Consumerist exposed the flog as the work of a PR agency.

This subject is examined in some detail in a paper entitled Strategic Manipulation of Internet Opinion Forums: Implications for Consumers and Firms by Chrysanthos Dellarocas (Sloan School of Management, Massachusetts Institute of Technology (2004).

The most striking result of their investigations was that strategic manipulations can either increase or decrease the information value of online forums to consumers relative to a case where no manipulations takes place. The study discovered circumstances in which the presence of honest consumer opinions induces firms to reveal their own, more precise knowledge of product qualities by manipulating the forums at intensities relative to the actual qualities of the product in question. However, if there are enough honest opinions, then manipulation can prove to be harmful to companies because costs outweighs benefits. Lastly, other channels of customer insight are not exempt from untruths – people can and do lie in online surveys, hotel quality questionnaires, telephone interviews and focus groups.

Many to many communication

Moreover, online sentiment is just one of several information sources consulted in pre-purchase consumer research. Even the most ardent word-of-mouth evangelist will not deny the influence of other channels on consumer behaviour and purchasing patterns. However, consumer-generated media provides qualitively distinct insight. Media content, company communications, advertisements are all essentially one to many communications. Online sentiment emerges from many to many discussions with points of view expressed, debated or even refuted by a wide range of opinion holders.

There are present customers talking to potential customers who are consulting former customers all of them tapping into opinion-leading blogs commenting on sector news flowing seamlessly between blogs, forums, offline conversations, e-mails, podcasts, social networks, news groups and traditional media in audiovisual, print and online formats.

Organizations might not be able to control what is said about them, their products, brands and services online. They can, however, at least begin to monitor (for potential threats and opportunities) and even take it one step further and extract valuable customer insight from it.

These are people expressing themselves spontaneously and of their own free will, which allows market research departments to discover new issues that were not included in questionnaires. A key value of this research is that is being freely expressed with no issue of question or researcher bias. Research bias can be very subtle such as the way a question is phrased if part of an online survey or asked if part of focus group research. For example, the question: What would you do if all the minibar drinks you drank were omitted from your hotel bill? This question would probably reveal many honest consumers in the presence of a researcher. Discussions in forums and blogs tend to be more candid as they are unprompted by a researcher. Specialist online sentiment monitoring software and analysts passively monitor the internet, never contributing or influencing the discussions.

Getting started

The start point for any research project is clearly establishing the objectives. Typically key issues to decide are time period, languages, key words or key concepts for searching social media for references This may be ongoing monitoring of brand and customer service perception for say, a hotel chain or monitoring awareness and favorability to a particular product before, during and after a particular campaign. Clients are interested in knowing which aspects of their products or services are discussed and how positively they are perceived.

The monitoring company may have a fairly good idea about this from previous experience and can set out to the client in the brief the key concepts and their respective sub concepts that make up online sentiment. In the case of an airline, for example, key concepts might be the following:

Tickets, Airport services, Airport staff, Boarding, Flight, On board services, Airplane staff, Baggage

Each of those concepts can be broken down into subconcepts. Taking On-board services as an example:

Food, Drinks, Service, Air conditioning, Blankets/pillows, Business class, Economy class, Coffee, In-flight entertainment, Lighting, Newspapers, Safety

The sentiment monitoring company needs to make it clear that it can only gather, analyse and present what people are discussing online. In some cases, a potential client may propose monitoring sentiment about a product, service or concept which generates very little in the way of discussion.

References to a product or a service vary a great deal in length and may contain a number of opinions on sub concepts ranging from one to a hundred or more This means that the analysts need to assign a positive/negative sentiment value at opinion level to truly understand which aspects are valued by consumers in social media. A reference can praise the way check-in staff handled overbooking by giving the passenger an upgrade to first class but then criticise the leg room and the dull food served on board.

Language, not country

A key issue for multinational clients to grasp is that online sentiment influence works mainly by language not by countries. Hence, opinion in English may be from a native speaker in Pittsburgh, London, Cape Town or Perth. Similarly, many people exchanging views online their second language (more often than not, English). French speaking Belgians participate in French forums, their Flemish-speaking counterparts in Dutch ones. Austrians will read German blogs, Germans will discuss issues in Swiss forums.

The applications of online sentiment intelligence cut across several functions within an organization: public relations/communications, marketing, customer relationship management, product/service development, stakeholder relations.
The insight can lead to modifying existing products, services, the way they are communicated and promoted, discovering new business opportunities and measuring the success of marketing and communications activities.

One size does not fit all

A particular project might be analysing sentiment about multiple product lines, in several languages with the intelligence to be used by multiple departments and across several territories.The board only needs topline statistics for the period. The PR manager’s insight needs differ from those of the product development department. Website managers want the feedback about user experiences on their websites. Marketers want immediate feedback on the product launch or to know which strengths to play to in their next camapign. Country managers might only be interested in sentiment in their respective languages. In some cases, feedback can get down to the business unit level – a particular store or hotel receiving topline statistics and verbatim references. At the other end of the scale, those in corporate social responsibility may want to discover how thier company is perceived by stakeholders as well as straight customers – labor organisations, lobby groups, pressure groups, the local community, employees.

Text and / or graphics reporting is limited by the fact that different people need different insight. As a result, it is essential that the data is presented in such a way that each person can hone in on what is useful to them, by using interactive viewing software that allows them to apply filters to the data and to drill down to the original references. Everyone may not have or indeed need access to this viewer software, hence those that have the software need to be able to export the data and graphics to conventional formats to produce their own reports. This is also important to be able to integrate online sentiment insight with online surveys, instore questionnaires, focus group findings.

Similarly, clients are likely to want to compare the study period in question (two weeks, month or quarter) with previous periods or see how an issue is developing cumulatively. They may even want to focus in on a particular week within a monthly report to see how consumers reacted to a particular campaign or piece of news. As a result, the ability to filter by date is also beneficial.

Typical filters normally include language, business line (for example hotel chains in a hotel group), opinion holder profile (customer, former customer, competitor customer, potential customer, employee, opinion-leader-journalist). This range of people sharing experiences online is one of the key strengths of social media sentiment insight in that it allows companies to access opinions from groups of people otherwise difficult to engage with. Former customers may be too angry to want anything to do with the source of their grievances. Potential customers and competitor customers are similarly difficult to access. On the downside,
blogging and participating in forums can be anonymous and so much of the information so vital to market researchers and their clients is often not available: sex, age, education, income group etc. Some of this information can be inferred by personal information on a blog (About Me) or from the text in a forum posting (“my teenage daughter thinks the dashboard is neat” – parent, probably in income group X that this car is aimed at, probably within age range 38 – 55).

In reference to the multi-functional applications of online sentiment insight, a filter defining the opinion type is often useful. Hence the analyst will categorise opinions according to whether they are declarations (statements about particular concepts), suggestions (“a rear windshield wiper would greatly improve the limited visibility of this car”), questions (“What’s the best way of getting from the station to this hotel?) or calls to action (i.e. recommendations “Don’t stay at this hotel, dirty rooms, rude staff and appalling food” or “I would definitely recommend a weekend stay in this fascinating town).

All of these opinions provide valuable insight but the latter three also provide an opportunity to engage with target audiences. Product development can get ideas for new features or even new products from suggestions. Travel forums are full of questions from people thinking about visiting city X or country Y. Tourist authorities are well-placed to answer these questions which are often answered by fellow travelers. Lastly, calls to action identify those people who are truly dissatisfied with the product or service and could form the basis for an outreach initiative in the customer service department to address issues directly with the person affected. The company may even consider participating in the debate so that other people reading it get a balanced view. Conversely, those recommending a product or service could be identified as key influencers to participate in a corporate blog or to collaborate as brand evangelists.

Client needs vary from organization to organization. One automotive client using an online sentiment insight service said: “the (reports) help you detect and track the hot issues and to see how the groups of people expressing opinions on the different models develop over time”. Knowing the perceived strengths and weaknesses of each model allows the client to design strategies and redefine messages aimed at accentuating the positive and mitigating the negative.

Another client from a well known soft drinks brand indicated that the company is “subject to numerous rumors and urban legends because of it’s brands reknown” Analysing online sentiment helps the client to discover if key messages are reaching their target audiences and to find out where people are talking about the company.

Listen, act and engage

The importance of online social media insight to an organization depends largely on the quantity of references to concepts such as its products, brands, services or people and the influence of these references on reputation and ultimately sales. For example, a group with several branded clothes store chains extracted significant insight about consumer perception of womens’ products, service and stores. The managers concerned then requested the same for some mens’ clothing chains only to discover that men discussed their clothes shopping experiences far less than women, at least online. In our experience, business to business product and services do not generate much CGM: companies do not go online to rant or rave about their suppliers. Airlines are sources of debate in CGM because we often have a choice of different airlines and seek to share our experiences of them. The airline we fly with usually determines the airports involved in our trip. Hence we see less utility in sharing our aiport experiences beyond factors involving the airline (check-in, customer service etc.) because, like us, our fellow travellers have often little choice in the airports they use.

Sentiment in online social media is just one of several elements determining how an organization, product, brand or service is perceived. Organisations should not throw the baby out with the bathwater – robust media monitoring and analysis will still be essential to many organizations as will other forms of customer intelligence – the myriad forms of quantitative and qualitative market research. However, the importance of social media intelligence in the media mix is set to increase, particularly as social networking sites grow and non-textual references (photos, video) increase. There are concepts which do not attract a significant amount of online comment at present but could do so in the future as social media evolves. In September 2007, UK-based confectionary group Cadbury announced the re-launch of its Wispa chocolate bar, defunct since 2003. A camapaign run on social network Facebook eventually had more than 14,000 users demanding the re-introduction of a 1980’s iconic brand. The story about the explosive mixture of a well-known soft drink and a candy brand spread principally through YouTube videos.

Some organisations are actively using social media for customer engagement. When JetBlue had to cancel or delay many New York flights over the Valentines weekend earlier this year due to bad weather, Chief Executive David Neeleman appeared on YouTube to reassure customers that the terrible service levels experienced would not be repeated. The company also made the effort to listen to as much customer feedback as possible. Consequently, customers posted positive messages on YouTube after Neeleman’s appearance pledging their support. As one customer commented on YouTube: “Honest communication is always appreciated even when things haven’t always been smooth.”

Do you speak digital?

Client needs vary from organization to organization With the media hype surrounding web 2.0 (and subsequent web 3.0, 4.0 etc.), many organizations are struggling to get to grips with an area many consider to be beyond control and consequently dangerous, irrelevant or both. Regardless of the type of organization, there is a permanent need to regularly review dedicating resources to listening to key audiences or stakeholders. Once those online conversations reach a certain level, regular monitoring becomes essential in to be less susceptible to suffering a web-spread reputation crisis. Once an organization is listening it can move on to acting upon the valuable intelligence that can be extracted from this dialogue and gain substantial competitive advantages.

A good place to start is by reviewing your current agency/supplier roster in market research, direct marketing,advertising, public relations and customer management? What ideas do they have for communicating and engaging with consumers online? Your suppliers need to become very good at online or if you are already working with online agencies, do they have the range of services and the planning discipline that more established “traditional”agencies have? Another recommended port of call is Social Target’s Guide to Social Media Analysis: a worldwide guide to the companies who monitor, measure and analyze online social media (www.socialtarget.com).

Online Qual Takes a Front Seat: Understanding Young Professionals for BalletMet

Steve Richardson — Fri, 12 Oct 2007 21:00:49 GMT

PDF version available here.

Abstract Qualitative research takes a front seat in Columbus, Ohio-based BalletMet’s effort to target young professionals. Participants kept a web-based experiential diary with comments and photos, answered questions via email, and engaged in web-based dialog to help uncover beliefs, motivations and feelings about the process of selecting entertainment options. Findings included young professionals’ desire to socialize and network at events, to have a well-rounded, multi-faceted experience, and to attend interactive and experiential entertainment. Moving forward, BalletMet is engaging crowds with interactive experiences, targeting “social coordinators” in groups, and offering affordable options for young professionals.

Author Information

Author1: Steve Richardson Affiliation: Qualitative Research Consultants Association Address: 1000 Westgate Drive, Suite 252 Address: St. Paul, MN 55114 USA Phone: 314-367-7359 Email: steve.richardson@qrca.org

The ability to extract quantitative research data using online tools is well-documented. Perhaps slightly less understood are the myriad of qualitative methodologies being conducted online that uncover deep-seated consumer insights and beliefs – often thought of as the “secret sauce” of a full-circle marketing research effort.

Qualitative research is designed to extract feelings, emotions and attitudes that motivate consumer behavior. Sometimes, it is the first step in the market research effort, followed by quantitative tools that seek to statistically measure those findings. Other times it is the flesh on the bones of a quantitative effort added after the basics are learned.

In the offline world, qualitative research is rich with a variety of techniques including focus groups, in-depth interviews, ethnography, projective techniques, storytelling, mapping, sentence completions and imagery. Some of these techniques translate well to the online world. In fact, focus groups conducted in the virtual world Second Life (www.secondlife.com) are getting much media attention these days. But some qualitative techniques are unique to the online world and offer researchers and buyers some advantages that no other medium offers.

Sheila Hunt, Director of Marketing Communications for BalletMet in Columbus, Ohio, experienced some of those advantages first-hand during an online research project designed to attract and retain young professionals to the ballet. She explains that BalletMet recognized that young professionals are the cultural art patrons of the future, as well as a large segment of the population, and BalletMet wanted to become more familiar with the interests and habits of this group. The goal was to help the organization develop programming that would appeal to young professionals ages 22-35.

“Conducting the qualitative research online was appealing, particularly with a younger demographic,” Hunt says. “We figured it would be fast, practical, and appealing to the participants, and we were right. This methodology was totally in their comfort zone.”

Trish Cadwallader, a researcher with Cincinnati-based Felton Willis and a member of the Qualitative Research Consultants Association (www.qrca.org), conducted an eight-day research project with young professionals this summer, all with an established interest in the cultural arts. Cadwallader used the “Revelation” web-based software to conduct the research, which was done entirely online. The software was developed and supported by Steve August, founder and CEO of Revelation, KDA Research based in Portland, Oregon, and also a member of QRCA.

During the research period, participants kept a web-based experiential diary, recording their exposure and selection of any type of entertainment. Participants discussed their entertainment choices and included photos from those events – everything from movie rentals at home to concerts, to “mojito night” with girlfriends. Participants also answered questions and engaged in a web-based dialog that helped uncover beliefs, motivations and feelings about the process of selecting entertainment options.

Cadwallader says one of the advantages of the software was her ability to “turn on” whether or not participants could view each other’s postings. Their individual diaries were shared only with the researcher, and Cadwallader could ask additional clarifying questions of participants throughout the course of the research.

In addition to individualized dialog, Cadwallader used an “open forum” where participants shared comments, answered questions, and reacted to each other’s postings. The forum was not live, so participants could go online and post whenever it was convenient.

“They were very involved on a daily basis, and I’m convinced part of the reason we got so much from them was because they could participate at their leisure,” Cadwallader says. “The dialog approach was particularly useful because that is where you start to uncover insights that you might not get from a one-time question or fill-in-the-blank type of approach.” She adds that participants were also entertained by each other’s photos and descriptions, which helped encourage participation.

The primary objectives of the research were to determine the drivers for selecting entertainment and how young professionals hear about events, understand who they attend events with and how far they plan ahead, gather feedback on their actual cultural experiences, and explore how best to connect with them in the future.

Many important insights came from the research, including what attracts young professionals to cultural events in the first place. They actively seek out events where other young professionals will be and where socializing and networking can take place. In addition to the obvious desire to experience entertainment where the subject matter is appealing, they also seek a well-rounded experience. For example, rather than just viewing an art exhibit, it is appealing to understand the artist’s life, motivations behind the art, and context in which the artist worked.

It was also clear to researchers that young professionals sought out interactive and experiential entertainment, as well as original programming or new twists on familiar programs. Several participants cited examples such as an Indian dance that took place during an intermission, and even tossing out T-shirts to the audience before performances to keep the atmosphere lively.

Also of use to BalletMet was the confirmation that getting young professionals to attend even one ballet performance typically changes the participants’ perception of what BalletMet offers. “Those who have not attended a performance since childhood think of pink tutus and The Nutcracker, but those attending as young adults are pleasantly surprised to see that we offer contemporary, edgy dance as well,” Cadwallader says. This knowledge, she says, reinforces BalletMet’s need to continue to target this group and create awareness, trial and repeat.

Other key takeaways from the research included: • Continue offering affordable options for young professionals. A program called “MetPass” lets young professionals receive up to two tickets per show at half price for a $20/year membership fee. Membership also includes discounts at restaurants and to other arts organizations. • Reach young professionals through email and where they gather and socialize. Hunt says BalletMet is evaluating the best tactics to reach them in social settings, but that email communication is up and running. • Engage the crowd with interactive experiences. Offering performances, giveaways and other activities before, during and after performances has been underway for a couple of years, but Hunt says that confirmation of attendees’ interest in these activities was one of the best takeaways of the research effort. • Target the “social coordinator” within a group of friends. In almost every group, Cadwallader says, there is a ringleader who guides entertainment decisions and purchases tickets for the group. Identifying and target those social coordinators with incentives is now being explored at BalletMet. • Make the whole experience an evening out, not just part of one. “Simple things like restaurant discounts and entertainment during intermissions provide patrons with a sense of having a full entertainment experience when they choose the ballet,” Hunt says.

Hunt adds that the online research effort achieved two important things – the confirmation that efforts already underway to target young professionals are being well-received, as well as insights that will help guide future marketing efforts toward groups like social coordinators. The research methodology, she adds, was critical to achieving these outcomes.

“This is how people as a culture are evolving,” Hunt says. “We have conducted focus groups and done hand-held surveys before – and we will continue to do so – but that research was done at our convenience. The online research efforts allowed our patrons to participate at their convenience, and I am convinced that we had better and more thorough participation for that reason.”

Steve Richardson is Director of Communications for the Qualitative Research Consultants Association. He can be reached at steve.richardson@qrca.org. For more information, visit www.qrca.org.

http://ijor.mypublicsquare.com/files/ijor/qualitatively/Columbus_Ballet_Case_Study_draft1.doc

More Honest Answers To Surveys? A Study of Data Collection Mode Effects

Mike Dennis, Rick Li — Wed, 10 Oct 2007 07:00:00 GMT

PDF version available here.

Abstract

This study is an attempt to contribute to previous research on the subject of data collection mode effects comparing specifically the Internet mode of data collection to telephone-based and in-person data collection. In this study, we controlled for sample source by having all interviews conducted with pre-recruited panelists from KnowledgePanelSM. The administered survey questions are the ‘national priority’ items from the General Social Survey (GSS). The actual GSS was in the field for the in-person data collection during the fielding of our experiment. In the analysis, we compared the results from the three modes of data collection – web, telephone, and in-person – to identify survey questions where the mode of data collection is related to a directional difference in the survey findings. While we found similarity in survey results for most items, the web mode appears to yield more honest and candid answering of sensitive questions about society.

Author Information

Author1: J. Michael Dennis
Affiliation: Senior Vice President, Government & Academic Research, Knowledge Networks
Address: 1350 Willow Road, Suite 102
Address: Menlo Park, CA 94025

Author2: Rick Li
Affiliation: Project Director, Government & Academic Research, Knowledge Networks
Address: 1350 Willow Road, Suite 102
Address: Menlo Park, CA 94025

Acknowledgement: We wish to thank Tom Smith and the staff of NORC who shared with us the question wording and results of the GSS national priority battery for the 2006 survey. Our conclusions and interpretations of the survey data are not intended to represent and should not be interpreted to represent the views of Mr. Smith or his colleagues at NORC.

Introduction

In 2000, NORC and Knowledge Networks (KN) conducted an experiment based on survey questions about national priorities from the General Social Survey (GSS), comparing KN panel results to those of the GSS for the same questions. In that experiment, the “Don’t Know” option was shown to respondents on their KN-provided WebTV screens. The results showed that respondents from the KN experiment were significantly more likely to indicate “Don’t Know” than the respondents from the in-person GSS (Smith, 2003). As a result of that (surprising) finding, NORC and KN in 2002 conducted another experiment as an extension to the year 2000 experiment to investigate the effects of data collection on survey responses. In that study, results from KnowledgePanelSM collected over the Internet via WebTV were compared to the results of the GSS survey conducted by in-person interviews. The results showed that when the “Don’t Know” option was not presented on-screen and respondents were instructed at the start of the survey to skip a question to indicate a “Don’t Know” response, the percentage of “Don’t Know” respondents in KN’s experiment was similar to that obtained in the in-person GSS (Smith and Dennis, 2005). At the time, this was a significant finding due to some concerns about the potential for high missing data rates for online surveys.

The study also showed that, with the exception of a few items that are possibly sensitive to social desirability bias, the differences on the substantive survey findings between KN and the GSS are fairly small. However, there was a pattern: Respondents to the in-person GSS survey, compared to the KN panelists, were consistently more likely to indicate that the country is spending “too little” on various national priorities and, for some priorities, the differences were large.

We were intrigued by these substantive differences between the two 2002 surveys. Specifically, in conducting the 2006 study, we sought to understand why the web panel respondents were more inclined to report that the country is doing “too much” to solve society’s problems while respondents asked questions by an in-person interviewer were more likely to say the opposite: namely, that the country is doing “too little” to solve society’s problems and presumably should be doing more. Web panelists, in general, did not feel as much urgency to solve some key societal problems, compared to respondents asked questions by interviewers visiting them in their homes. We wondered whether the substantive differences are the result of underlying differences in the representativeness of the survey samples or, more interestingly, are the differences a reflection of the fact that one mode of data collection employs interviewers and the other one does not. In other words, are the differences the result of survey sampling or the result of different modes of data collection – “interviewer-based” versus “interviewer-less” surveys?

To answer this question, the current study added an experimental component, following the model of other KN methodological work on data collection mode effects (Dennis et al, 2005). We added one mode of data collection to the design—the telephone—while keeping the sample source the same.. Specifically, KN administered the same GSS questionsto two independent, nationally representative samples drawn from the same KnowledgePanelSM. One KN panel sample was drawn for administration of the survey by web – as is customary for KN – while the other KN panel sample was administered the same survey by telephone interviewers. Obviously, the two samples do not overlap. As in the year 2000 and 2002 surveys, the questionnaire was primarily composed of the national priority spending questions from the latest General Social Survey (GSS). Also, as in the year 2000 and 2002 studies, the KN panel survey was conducted during the same time that the GSS was conducted by personal interviewers (NORC). By this design, we are able to isolate the effects of survey sample source versus the mode of data collection. This is not an academic exercise: we researchers are often asked by our customers if moving a survey to the Internet will “change the results.”

In addition to this research question, we also sought to re-examine the issue of whether the Internet mode of data collection produces more “don’t know” responses than other modes of data collection. Despite our past research on this topic, we are still asked occasionally whether the web panel data will engender more missing data than a comparable telephone or in-person survey, and for this reason, we retained this part of our research agenda.

Methods

The online mode of data collection began on March 29, 2006 and ended on May 15, 2006. A total of 1,689 KnowledgePanelSM panelists were invited to participate in the survey and 1,428 (84%) completed the survey. No study-specific respondent incentives were provided. The survey completion rate was relatively high because of the long field period and due to the use of Knowlede PanelSM. The “Don’t Know” option was not shown on the screen. Respondents were given an instruction at the beginning of the survey that they should skip the question to indicate “Don’t Know” or “No opinion.” This treatment of the “Don’t Know” response option was tested and previously reported (Smith and Dennis, 2005).

The phone mode of data collection underwent two stages. First, due to the agreement between KN and its panelists, an explicit consent from the respondent had to be obtained before a phone survey could be initiated. Between May 4, 2006 and June 30, 2006, a total of 1,383 KnowledgePanelSM panelists were asked whether they would be interested in participating in a short survey on the phone. Of the 1,208 (87%) panelists who responded, 839 (70%) agreed to participate in the phone survey. The consented panelists were subsequently contacted for the phone survey. In the end, 600 (71%) phone interviews were completed, and each case received $10 as an incentive.

The following table summarizes the three modes of data collection:

Table 2: Summary of Modes of Data Collection

Table 3 shows the unweighted demographic characteristics of those who completed the online and the phone surveys, as compared with the U.S. Census Current Population Survey (CPS) benchmarks. With a few exceptions, the respondents of the two modes of data collection are similar to each other. Phone respondents are slightly underrepresented in men, younger, less educated, and ethnic minorities. However, the respondents of the online mode are more similar to the CPS benchmarks than the respondents of the phone mode.

Table 3: Sample Demographics (Unweighted)

Note: The Internet access rates for the KN panelists are based on Internet status as of the time of recruitment into the KN panel.

Analysis

All of the analyses presented in this paper are weighted using post-stratification weights that incorporate the probabilities of selection. Both the phone and online mode respondents are weighted to the latest CPS benchmarks.

We first explore once more the issue of “Don’t know” rates that were obtained by web versus interviewer-administered surveys, and then proceed to a discussion of systematic differences seen in the survey results.

“Don’t Know” Rates

Table 4 shows the percentage of respondents who indicated “Don’t
Know” in KN’s experiments in 2002 and 2006, compared with the 2002 and 2006 GSS
results. The “Don’t Know” rates of GSS stayed fairly the same between 2002 and 2006,
while the “Don’t Know” rates of KN’s online experiments showed a slight decline from
2002 to 2006. For 15 of the 17 spending items, this decline was between 1-2 percentage points. For the remaining two spending items, the decline was 3-4 percentage points.

Despite these small declines, the average “Don’t Know” rates remained fairly similar
across the different experiments in these two years. The average “Don’t Know” rates for
the in-person GSS were around 4%, compared to an average of 2%–3.5% of KN’s online and phone experiments.

These data show a successful replication of the 2002 findings. By not displaying the “Don’t Know” on screen but instructing respondents to skip a question to indicate “Don’t Know,” we observed “Don’t Know” rates similar to those obtained from the in-person survey for most spending items.

Table 4: “Don’t Know” Rates in 2002 and 2006

Substantive Findings

Respondents were presented with a list of 17 national spending priority items and were asked whether they felt “we were spending too much, too little, or about the right amount” on each item.

Figure 1 shows the percentage of respondents who think we are spending “too little” on each of the national priorities. The x-axis shows the percentage of GSS in-person respondents answering that we are spending “too little” on national priorities. The blue diamond symbols show the comparison between the GSS and phone results, while the red squares show the comparison between the GSS and Internet results. The distance from the diagonal line shows the magnitude of the differences between the results. The farther the symbols are from the diagonal line, the more different the results are from the GSS in-person survey. Should the three modes produce identical results, the symbols would align perfectly on the diagonal line. If a symbol lies above the diagonal line, more people from the Internet or phone mode than the in-person mode indicated we are spending “too little” on the national priority item. In contrast, if a symbol lies below the diagonal line, more people from the GSS in-person survey than the Internet or phone mode indicated we are spending “too little” on the national priority item. For example, on “Improving the condition of Blacks,” 36% of GSS in-person respondents indicated we are spending “too little” while only 26% of the online respondents answered this way.

The results from the GSS in-person and the telephone mode are very similar, as seen in how the diamond symbols are generally located close to the diagonal line. In contrast, as shown by how the square symbols are generally located below the diagonal line, respondents from the in-person GSS survey and KN’s phone survey are consistently more likely to say we are spending “too little” on the national priorities. The surveys administered by interviewers tended to produce results suggesting that the country should do more to solve society’s problems. Although the phone and Internet modes have the same sample source, the results from the phone mode are more similar to the results from the in-person mode than to the Internet mode.

Some of the national priorities that, in our view, are the most sensitive and potentially controversial are also the ones where there are the largest substantive differences. The spending priorities with the largest differences between the in-person and online modes are these: “Dealing with drug addiction” (14 percentage points), “Improving the conditions of Blacks” (11 percentage points), “Solving the problems with big cities” (10 percentage points), “Assistance for childcare” (10 percentage points), and “Supporting scientific research” (10 percentage points).

Figure 1: Scatter plots of “Too Little” Respondents

Figure 2, a partial mirror image of Figure 1, shows the percentage of respondents who think we are spending “too much” on each of the national priorities. Again, the x-axis shows the percentage of respondents from the 2006 GSS in-person survey. The blue diamond symbols show the comparison between the GSS in-person and the phone survey, while the red square-shaped symbols show the comparison between the GSS in-person and the Internet survey. If a symbol lies above the diagonal line, more people from the Internet or phone mode than the in-person mode indicated “too much” on the national priority item represented by this symbol.

The results on the spending “too much” responses are nearly a reverse image of the “too little” responses. Again, the results from the GSS in-person mode are very similar to those from the KN phone mode. Again, the interviewer-less mode produces distinctive results: Respondents from the Internet mode are consistently more likely to say we are spending “too much” on the national priorities than are respondents from the in-person and phone modes. The largest differences are found for the priorities that might be considered the most controversial or sensitive: “Improving the conditions of Blacks” (18 percentage points), “Foreign aid” (8 percentage points), “The space exploration program” (7 percentage points), and “Welfare” (7 percentage points). A public policy informed only by the in-person survey would suggest that the polity needs to do more on these issues, compared to a public policy informed by the web panel survey.

Figure 2: Scatter plots of “Too Much” Respondents

&amp;amp;lt;img src="http://ijor.mypublicsquare.com/files/ijor/more-honest-answers/kn_too_much.png"&amp;amp;gt;

Table 5 displays the detailed substantive findings from the three modes of data collection. The first block of numbers shows the percentage of respondents indicating we are spending “too much”, “too little”, and “about the same” from the GSS in-person survey. The next block of numbers shows the differences between the online and in-person modes in 2002. The next two blocks show the differences between the GSS in-person survey and KN’s online and phone modes from 2006. A positive number indicates that more respondents selected the response option in person than on the phone or online, and a negative number indicates that more respondents selected the response on the phone or online than in person.

The average absolute difference in survey results between the GSS in-person survey and the phone mode is only 2 percentage points. The average absolute difference between GSS the in-person mode and the online mode is between 4 and 6 percentage points in 2006, which is similar to the average absolute difference of four percentage points we observed in the 2002 study. The shaded rows in the table highlight some of the spending items on which the differences between the in-person and online modes were replicated in very similar magnitude.

While the substantive differences in the aggregate are not enormous, there is a grouping of differences that suggest to us that respondents are being more honest or more willing to risk reporting a controversial opinion about how much the country should do about competing national priorities.

Table 5: Substantive Results from In-Person GSS and KN Online and Phone Experiments in 2002 and 2006

* The shaded rows indicate the spending items on which the difference between the in-person and online mode was replicated in both 2002 and 2006.

Discussion and Conclusion

This paper continues and expands the research on modes of data collection in 2000 and 2002 using the GSS national priority item battery. In 2002, we compared KN’s online experimental treatments of “Don’t Know” responses with the actual in-person GSS results. In the current research, we examined the differences between the online, phone, and in-person modes on 17 national spending priority items from the General Social Survey. The results suggest the following findings:

• The results from the year 2002 study were replicated:

– By not showing the “Don’t Know” option on screen but instructing respondents to skip the question to indicate “Don’t Know,” KN’s online surveys have a “Don’t Know” rate similar to those obtained from,in-person and telephone surveys. This is a non-trivial finding for analysts who have concerns that online data collection necessarily implies a high rate of missing data. This finding also contradicts the justification for the procedure employed by some survey research firms that take the form of “forcing” an answer from respondents. Our experiment shows that relatively low missing data rates can be obtained without either forcing a response or by conspiculously showing a “Don’t Know” response option on the screen.

– Similar to the finding in the 2002 study, respondents from KN’s online survey are consistently less likely to select the option that the country is spending “too little” on selected national priorities and more likely to select spending “too much, ” compared to respondents participating in the in-person survey.

– In the 2002 study, spending items dealing with the urban underclass (Blacks, big cities, crimes, drugs, and welfare) and foreign aid showed large differences between the online and in-person modes. These large differences continued in the 2006 study for the same spending priority items.

• The fact that the phone and online modes have the same sampling source (i.e., KnowledgePanelSM) did not predetermine the similarities in the results between these two modes. To the contrary, the dissimilarity in results between the phone and online modes and the similarities between the phone and in-person modes are suggestive of strong effects due to varying the mode of data collection:

– The average difference between the in-person and phone modes is smaller than the average difference between the in-person and online modes.

– The systematic differences between the in-person and online modes do not exist between the in-person and phone modes.

– The large differences between the in-person and online modes on the spending items dealing with urban social underclass and foreign aid decreased or disappeared completely between the in-person and phone modes.

These observations lead us to conclude that there are important differences in the survey results that are attributable to the presence of an interviewer for the in-person and telephone modes, and to the absence of an interviewer in the web mode. The direction of the differences in the survey results, as seen in how respondents are more likely to report in the web mode that the country spends “too much” on certain problems in society, is consistent with the conclusion that web panel respondents are more honest and exhibit more candor in their responses, compared to interviewer-administered surveys. This conclusion is reinforced by the experimental design of our study, which controlled for the source of the sample. To be clear, we are not indicating that we know the “true” measure for public opinion, nor are we suggesting that the online mode survey results are closer to the “truth” about U.S. public opinion. However, we do believe that the differences we observe in the survey results are consistent with the hypothesis that online respondents feel less potent pressure to answer questions in socially desirable ways.

References

Dennis, J. M., Chatt, C., Li, R., Motta-Stanko, A., and Pulliam, P. 2005. Data Collection Mode Effects Controlling for Sample Origins in a Panel Survey: Telephone versus Internet. Paper presented at the 2005 Annual Meeting of the American Association for Public Opinion Research. Available at: http://www.knowledgenetworks.com/ganp/reviewer-info.html

Smith, T.W. 2003, An experimental comparison of Knowledge Networks and the GSS, International Journal of Public Opinion Research 15, 167–179.

Smith, T. and Dennis, J.M. 2005, Online vs. In-Person: Experiments with Mode, Format, and Question Wordings, Public Opinion Pros: http://www.publicopinionpros.com/from_field/2005/dec/smithkn.asp

Smith, T.W. 2007, Trends in National Spending Priorities, 1973–2006, Chicago: NORC: http://www.norc.org/projects/General+Social+Survey.htm