More Honest Answers To Surveys? A Study of Data Collection Mode Effects

PDF version available here.

Abstract

This study is an attempt to contribute to previous research on the subject of data collection mode effects comparing specifically the Internet mode of data collection to telephone-based and in-person data collection. In this study, we controlled for sample source by having all interviews conducted with pre-recruited panelists from KnowledgePanelSM. The administered survey questions are the ‘national priority’ items from the General Social Survey (GSS). The actual GSS was in the field for the in-person data collection during the fielding of our experiment. In the analysis, we compared the results from the three modes of data collection – web, telephone, and in-person – to identify survey questions where the mode of data collection is related to a directional difference in the survey findings. While we found similarity in survey results for most items, the web mode appears to yield more honest and candid answering of sensitive questions about society.

Author Information

Author1: J. Michael Dennis
Affiliation: Senior Vice President, Government & Academic Research, Knowledge Networks
Address: 1350 Willow Road, Suite 102
Address: Menlo Park, CA 94025

Author2: Rick Li
Affiliation: Project Director, Government & Academic Research, Knowledge Networks
Address: 1350 Willow Road, Suite 102
Address: Menlo Park, CA 94025

Acknowledgement: We wish to thank Tom Smith and the staff of NORC who shared with us the question wording and results of the GSS national priority battery for the 2006 survey. Our conclusions and interpretations of the survey data are not intended to represent and should not be interpreted to represent the views of Mr. Smith or his colleagues at NORC.

Introduction

In 2000, NORC and Knowledge Networks (KN) conducted an experiment based on survey questions about national priorities from the General Social Survey (GSS), comparing KN panel results to those of the GSS for the same questions. In that experiment, the “Don’t Know” option was shown to respondents on their KN-provided WebTV screens. The results showed that respondents from the KN experiment were significantly more likely to indicate “Don’t Know” than the respondents from the in-person GSS (Smith, 2003). As a result of that (surprising) finding, NORC and KN in 2002 conducted another experiment as an extension to the year 2000 experiment to investigate the effects of data collection on survey responses. In that study, results from KnowledgePanelSM collected over the Internet via WebTV were compared to the results of the GSS survey conducted by in-person interviews. The results showed that when the “Don’t Know” option was not presented on-screen and respondents were instructed at the start of the survey to skip a question to indicate a “Don’t Know” response, the percentage of “Don’t Know” respondents in KN’s experiment was similar to that obtained in the in-person GSS (Smith and Dennis, 2005). At the time, this was a significant finding due to some concerns about the potential for high missing data rates for online surveys.

The study also showed that, with the exception of a few items that are possibly sensitive to social desirability bias, the differences on the substantive survey findings between KN and the GSS are fairly small. However, there was a pattern: Respondents to the in-person GSS survey, compared to the KN panelists, were consistently more likely to indicate that the country is spending “too little” on various national priorities and, for some priorities, the differences were large.

We were intrigued by these substantive differences between the two 2002 surveys. Specifically, in conducting the 2006 study, we sought to understand why the web panel respondents were more inclined to report that the country is doing “too much” to solve society’s problems while respondents asked questions by an in-person interviewer were more likely to say the opposite: namely, that the country is doing “too little” to solve society’s problems and presumably should be doing more. Web panelists, in general, did not feel as much urgency to solve some key societal problems, compared to respondents asked questions by interviewers visiting them in their homes. We wondered whether the substantive differences are the result of underlying differences in the representativeness of the survey samples or, more interestingly, are the differences a reflection of the fact that one mode of data collection employs interviewers and the other one does not. In other words, are the differences the result of survey sampling or the result of different modes of data collection – “interviewer-based” versus “interviewer-less” surveys?

To answer this question, the current study added an experimental component, following the model of other KN methodological work on data collection mode effects (Dennis et al, 2005). We added one mode of data collection to the design—the telephone—while keeping the sample source the same.. Specifically, KN administered the same GSS questionsto two independent, nationally representative samples drawn from the same KnowledgePanelSM. One KN panel sample was drawn for administration of the survey by web – as is customary for KN – while the other KN panel sample was administered the same survey by telephone interviewers. Obviously, the two samples do not overlap. As in the year 2000 and 2002 surveys, the questionnaire was primarily composed of the national priority spending questions from the latest General Social Survey (GSS). Also, as in the year 2000 and 2002 studies, the KN panel survey was conducted during the same time that the GSS was conducted by personal interviewers (NORC). By this design, we are able to isolate the effects of survey sample source versus the mode of data collection. This is not an academic exercise: we researchers are often asked by our customers if moving a survey to the Internet will “change the results.”

In addition to this research question, we also sought to re-examine the issue of whether the Internet mode of data collection produces more “don’t know” responses than other modes of data collection. Despite our past research on this topic, we are still asked occasionally whether the web panel data will engender more missing data than a comparable telephone or in-person survey, and for this reason, we retained this part of our research agenda.

Methods

The online mode of data collection began on March 29, 2006 and ended on May 15, 2006. A total of 1,689 KnowledgePanelSM panelists were invited to participate in the survey and 1,428 (84%) completed the survey. No study-specific respondent incentives were provided. The survey completion rate was relatively high because of the long field period and due to the use of Knowlede PanelSM. The “Don’t Know” option was not shown on the screen. Respondents were given an instruction at the beginning of the survey that they should skip the question to indicate “Don’t Know” or “No opinion.” This treatment of the “Don’t Know” response option was tested and previously reported (Smith and Dennis, 2005).

The phone mode of data collection underwent two stages. First, due to the agreement between KN and its panelists, an explicit consent from the respondent had to be obtained before a phone survey could be initiated. Between May 4, 2006 and June 30, 2006, a total of 1,383 KnowledgePanelSM panelists were asked whether they would be interested in participating in a short survey on the phone. Of the 1,208 (87%) panelists who responded, 839 (70%) agreed to participate in the phone survey. The consented panelists were subsequently contacted for the phone survey. In the end, 600 (71%) phone interviews were completed, and each case received $10 as an incentive.

The following table summarizes the three modes of data collection:

Table 2: Summary of Modes of Data Collection

Table 3 shows the unweighted demographic characteristics of those who completed the online and the phone surveys, as compared with the U.S. Census Current Population Survey (CPS) benchmarks. With a few exceptions, the respondents of the two modes of data collection are similar to each other. Phone respondents are slightly underrepresented in men, younger, less educated, and ethnic minorities. However, the respondents of the online mode are more similar to the CPS benchmarks than the respondents of the phone mode.

Table 3: Sample Demographics (Unweighted)

Note: The Internet access rates for the KN panelists are based on Internet status as of the time of recruitment into the KN panel.

Analysis

All of the analyses presented in this paper are weighted using post-stratification weights that incorporate the probabilities of selection. Both the phone and online mode respondents are weighted to the latest CPS benchmarks.

We first explore once more the issue of “Don’t know” rates that were obtained by web versus interviewer-administered surveys, and then proceed to a discussion of systematic differences seen in the survey results.

“Don’t Know” Rates

Table 4 shows the percentage of respondents who indicated “Don’t
Know” in KN’s experiments in 2002 and 2006, compared with the 2002 and 2006 GSS
results. The “Don’t Know” rates of GSS stayed fairly the same between 2002 and 2006,
while the “Don’t Know” rates of KN’s online experiments showed a slight decline from
2002 to 2006. For 15 of the 17 spending items, this decline was between 1-2 percentage points. For the remaining two spending items, the decline was 3-4 percentage points.

Despite these small declines, the average “Don’t Know” rates remained fairly similar
across the different experiments in these two years. The average “Don’t Know” rates for
the in-person GSS were around 4%, compared to an average of 2%–3.5% of KN’s online and phone experiments.

These data show a successful replication of the 2002 findings. By not displaying the “Don’t Know” on screen but instructing respondents to skip a question to indicate “Don’t Know,” we observed “Don’t Know” rates similar to those obtained from the in-person survey for most spending items.

Table 4: “Don’t Know” Rates in 2002 and 2006

Substantive Findings

Respondents were presented with a list of 17 national spending priority items and were asked whether they felt “we were spending too much, too little, or about the right amount” on each item.

Figure 1 shows the percentage of respondents who think we are spending “too little” on each of the national priorities. The x-axis shows the percentage of GSS in-person respondents answering that we are spending “too little” on national priorities. The blue diamond symbols show the comparison between the GSS and phone results, while the red squares show the comparison between the GSS and Internet results. The distance from the diagonal line shows the magnitude of the differences between the results. The farther the symbols are from the diagonal line, the more different the results are from the GSS in-person survey. Should the three modes produce identical results, the symbols would align perfectly on the diagonal line. If a symbol lies above the diagonal line, more people from the Internet or phone mode than the in-person mode indicated we are spending “too little” on the national priority item. In contrast, if a symbol lies below the diagonal line, more people from the GSS in-person survey than the Internet or phone mode indicated we are spending “too little” on the national priority item. For example, on “Improving the condition of Blacks,” 36% of GSS in-person respondents indicated we are spending “too little” while only 26% of the online respondents answered this way.

The results from the GSS in-person and the telephone mode are very similar, as seen in how the diamond symbols are generally located close to the diagonal line. In contrast, as shown by how the square symbols are generally located below the diagonal line, respondents from the in-person GSS survey and KN’s phone survey are consistently more likely to say we are spending “too little” on the national priorities. The surveys administered by interviewers tended to produce results suggesting that the country should do more to solve society’s problems. Although the phone and Internet modes have the same sample source, the results from the phone mode are more similar to the results from the in-person mode than to the Internet mode.

Some of the national priorities that, in our view, are the most sensitive and potentially controversial are also the ones where there are the largest substantive differences. The spending priorities with the largest differences between the in-person and online modes are these: “Dealing with drug addiction” (14 percentage points), “Improving the conditions of Blacks” (11 percentage points), “Solving the problems with big cities” (10 percentage points), “Assistance for childcare” (10 percentage points), and “Supporting scientific research” (10 percentage points).

Figure 1: Scatter plots of “Too Little” Respondents

Figure 2, a partial mirror image of Figure 1, shows the percentage of respondents who think we are spending “too much” on each of the national priorities. Again, the x-axis shows the percentage of respondents from the 2006 GSS in-person survey. The blue diamond symbols show the comparison between the GSS in-person and the phone survey, while the red square-shaped symbols show the comparison between the GSS in-person and the Internet survey. If a symbol lies above the diagonal line, more people from the Internet or phone mode than the in-person mode indicated “too much” on the national priority item represented by this symbol.

The results on the spending “too much” responses are nearly a reverse image of the “too little” responses. Again, the results from the GSS in-person mode are very similar to those from the KN phone mode. Again, the interviewer-less mode produces distinctive results: Respondents from the Internet mode are consistently more likely to say we are spending “too much” on the national priorities than are respondents from the in-person and phone modes. The largest differences are found for the priorities that might be considered the most controversial or sensitive: “Improving the conditions of Blacks” (18 percentage points), “Foreign aid” (8 percentage points), “The space exploration program” (7 percentage points), and “Welfare” (7 percentage points). A public policy informed only by the in-person survey would suggest that the polity needs to do more on these issues, compared to a public policy informed by the web panel survey.

Figure 2: Scatter plots of “Too Much” Respondents

<img src="http://ijor.mypublicsquare.com/files/ijor/more-honest-answers/kn_too_much.png">

Table 5 displays the detailed substantive findings from the three modes of data collection. The first block of numbers shows the percentage of respondents indicating we are spending “too much”, “too little”, and “about the same” from the GSS in-person survey. The next block of numbers shows the differences between the online and in-person modes in 2002. The next two blocks show the differences between the GSS in-person survey and KN’s online and phone modes from 2006. A positive number indicates that more respondents selected the response option in person than on the phone or online, and a negative number indicates that more respondents selected the response on the phone or online than in person.

The average absolute difference in survey results between the GSS in-person survey and the phone mode is only 2 percentage points. The average absolute difference between GSS the in-person mode and the online mode is between 4 and 6 percentage points in 2006, which is similar to the average absolute difference of four percentage points we observed in the 2002 study. The shaded rows in the table highlight some of the spending items on which the differences between the in-person and online modes were replicated in very similar magnitude.

While the substantive differences in the aggregate are not enormous, there is a grouping of differences that suggest to us that respondents are being more honest or more willing to risk reporting a controversial opinion about how much the country should do about competing national priorities.

Table 5: Substantive Results from In-Person GSS and KN Online and Phone Experiments in 2002 and 2006

* The shaded rows indicate the spending items on which the difference between the in-person and online mode was replicated in both 2002 and 2006.

Discussion and Conclusion

This paper continues and expands the research on modes of data collection in 2000 and 2002 using the GSS national priority item battery. In 2002, we compared KN’s online experimental treatments of “Don’t Know” responses with the actual in-person GSS results. In the current research, we examined the differences between the online, phone, and in-person modes on 17 national spending priority items from the General Social Survey. The results suggest the following findings:

• The results from the year 2002 study were replicated:

– By not showing the “Don’t Know” option on screen but instructing respondents to skip the question to indicate “Don’t Know,” KN’s online surveys have a “Don’t Know” rate similar to those obtained from,in-person and telephone surveys. This is a non-trivial finding for analysts who have concerns that online data collection necessarily implies a high rate of missing data. This finding also contradicts the justification for the procedure employed by some survey research firms that take the form of “forcing” an answer from respondents. Our experiment shows that relatively low missing data rates can be obtained without either forcing a response or by conspiculously showing a “Don’t Know” response option on the screen.

– Similar to the finding in the 2002 study, respondents from KN’s online survey are consistently less likely to select the option that the country is spending “too little” on selected national priorities and more likely to select spending “too much, ” compared to respondents participating in the in-person survey.

– In the 2002 study, spending items dealing with the urban underclass (Blacks, big cities, crimes, drugs, and welfare) and foreign aid showed large differences between the online and in-person modes. These large differences continued in the 2006 study for the same spending priority items.

• The fact that the phone and online modes have the same sampling source (i.e., KnowledgePanelSM) did not predetermine the similarities in the results between these two modes. To the contrary, the dissimilarity in results between the phone and online modes and the similarities between the phone and in-person modes are suggestive of strong effects due to varying the mode of data collection:

– The average difference between the in-person and phone modes is smaller than the average difference between the in-person and online modes.

– The systematic differences between the in-person and online modes do not exist between the in-person and phone modes.

– The large differences between the in-person and online modes on the spending items dealing with urban social underclass and foreign aid decreased or disappeared completely between the in-person and phone modes.

These observations lead us to conclude that there are important differences in the survey results that are attributable to the presence of an interviewer for the in-person and telephone modes, and to the absence of an interviewer in the web mode. The direction of the differences in the survey results, as seen in how respondents are more likely to report in the web mode that the country spends “too much” on certain problems in society, is consistent with the conclusion that web panel respondents are more honest and exhibit more candor in their responses, compared to interviewer-administered surveys. This conclusion is reinforced by the experimental design of our study, which controlled for the source of the sample. To be clear, we are not indicating that we know the “true” measure for public opinion, nor are we suggesting that the online mode survey results are closer to the “truth” about U.S. public opinion. However, we do believe that the differences we observe in the survey results are consistent with the hypothesis that online respondents feel less potent pressure to answer questions in socially desirable ways.

References

Dennis, J. M., Chatt, C., Li, R., Motta-Stanko, A., and Pulliam, P. 2005. Data Collection Mode Effects Controlling for Sample Origins in a Panel Survey: Telephone versus Internet. Paper presented at the 2005 Annual Meeting of the American Association for Public Opinion Research. Available at: http://www.knowledgenetworks.com/ganp/reviewer-info.html

Smith, T.W. 2003, An experimental comparison of Knowledge Networks and the GSS, International Journal of Public Opinion Research 15, 167–179.

Smith, T. and Dennis, J.M. 2005, Online vs. In-Person: Experiments with Mode, Format, and Question Wordings, Public Opinion Pros: http://www.publicopinionpros.com/from_field/2005/dec/smithkn.asp

Smith, T.W. 2007, Trends in National Spending Priorities, 1973–2006, Chicago: NORC: http://www.norc.org/projects/General+Social+Survey.htm