Home | Business News | Browse by Publication | I | International Journal of Market Research

Measuring the attitudes of the general public via internet polls: an evaluation.

Article Excerpt
Internet polls based on volunteer panels have quickly captured a significant slice of the UK polling market, based in large part on success at predicting the outcome of recent elections. However, opinion research is most usually conducted on a wide range of issues that cannot be measured against an election outcome and are only loosely linked to voting behaviour. This paper compares the results obtained from a representative sample of people interviewed by telephone with the internet-accessible population, those willing to join an internet panel and those who actually respond online. Across a range of subjects similar results are obtained, but on others differences emerge that suggest online panels cannot reliably replicate results obtained by more traditional research methods.

Introduction

The rapid diffusion of the internet in recent years has brought in its wake a new method of conducting opinion polls. In the UK this new approach has been championed in particular by YouGov, which in just two years has rapidly captured a significant proportion of the market. The company has done so not least because of its success at predicting the outcome of a number of recent events, including the 2002 London borough elections and the 2003 Scottish Parliamentary election, as well as the result of the Conservative leadership election and the first Pop Idol contest. Given the continuing difficulties that some polling companies have had in providing an accurate picture of vote intentions at elections (Crewe 2001), YouGov's recent record has proved to be a formidable selling-point.

This success has of course been achieved in the face of most conventional wisdom. YouGov's polls are not probability samples. Each YouGov poll is conducted among a subset of a panel of people who have visited the YouGov website and signed up to participate in the company's surveys. They are thus similar in character to the polls conducted in the US by Harris Interactive (Taylor 2000). The method means that those who do not have access to the internet (who still constitute around half of the UK population) and those internet users who have not enrolled to participate in YouGov's polls have no chance of being polled by the company.

But while success in predicting the outcome of elections may be necessary evidence of the ability of internet polling to provide an accurate picture of public attitudes, it is not on its own sufficient as most polls are conducted on subjects unconnected with vote intentions. Carefully designed and controlled comparisons between internet and probability samples in the US (where internet penetration is 50% higher than in the UK) have indicated that sometimes the two methods produce different results (Krosnick & Chang 2001; Smith 2003). So in this paper the authors report the results of research they have conducted in the UK to examine whether the results of internet polls based on a volunteer panel can be relied upon to provide accurate estimates of the distribution of public opinion as well as vote intentions. The key finding is that Couper's (2000) comment that 'the extent to which weighting the results from individual panels can reliably produce reasonable estimates is unknown' still has considerable validity.

The characteristics of internet polling

There are at least five characteristics of internet polling of the kind conducted by YouGov that might lead to results that are different from those obtained by other, more conventional methods such as telephone interviews undertaken using random digit dialling, the method that in the 1990s became the most common in conventional UK opinion polling.

Sampling bias

Not everyone has access to the internet. As already noted, only 50% or so of the UK population currently have access, and those who do tend to come from the younger and more affluent sections of the population.

Sampling frame

Probability sampling requires a sampling frame (such as the electoral register or postal address file) that makes it possible to undertake a random selection of persons to interview. No sampling frame of persons in the general population who have access to the internet is available. There are a number of ways of meeting this difficulty (Couper 2000). The one with which this paper is concerned is a non-probability method of recruiting a panel of volunteers via advertisements on selected websites, which is the method used by YouGov.

Response rates

While all modes of survey research have experienced declining response rates in recent years, most researchers who have conducted internet surveys in parallel with other methods have found that the response rates to internet surveys are usually lower than for other methods (Cook et al. 2000; Krosnick & Chang 2001; Vehovar 2003).

Conditioning and attrition

Those who are asked on a regular basis for their views are always liable to have those views influenced by the very fact that their opinions are regularly being solicited (Mitofsky 1999). Meanwhile, those who decide to maintain their membership of an internet panel may well be different from those who at some point opt to drop out (Gregory 2002)

Mode effects

The answers that people give to pollsters may well depend on how they are interviewed. It has, inter alia, been suggested that people are more likely to answer sensitive questions honestly in an online poll as there is no requirement to give a potentially embarrassing answer to an interviewer, that they are more likely to use the end-points of a five-point scale and that they are more likely to give apparently incorrect answers. However, the extent and even the consistency of these effects is far from clear (Taylor 2000; Krosnick & Chang 2001; Vehovar & Lozar Manfreda 2002; Smith 2003).

The focus in this paper is primarily on the possible impact of the first three of these characteristics. The authors are concerned to establish the extent of the potential bias that might be introduced into a YouGov-type internet poll by the fact that half the population is excluded, that people have to volunteer to participate and that even many of those who do volunteer may fail to complete any individual poll. In so doing the authors bear in mind the possible existence of mode effects, but their research does not encompass the possible impact of conditioning and attrition. This exclusion would not, however, appear to vitiate the conclusions that are drawn.

Data

Between October 2002 and August 2003 ICM set out to recruit a panel of people willing to join an internet polling panel via the company's telephone omnibus survey. Apart from basic demographic questions respondents were asked whether they had access to the internet and whether or not they would be prepared to join an internet polling panel (after being offered an incentive similar to that offered by YouGov of [pounds sterling]0.50 for every survey completed). In all, 71,018 interviews were contacted--a response rate of 25%. Of that total 61% (43,141 people) claimed to have access to the internet and 19% (13,501 people) agreed to join an internet polling panel. We can use these data to compare the demographic characteristics of, first, those who are accessible over the internet and, second, those who say they are willing to join an internet panel, with the characteristics of those who respond to a telephone poll. Meanwhile, the demographic representativeness of this telephone poll can be assessed by comparing its data with those collected by a random face-to-face probability sample, the National Readership Survey (NRS). (1)

In addition, the first 4014 people who were contacted as part of the panel building programme, that is those...

View this article FREE - Now for a Limited Time, try Goliath Business News
Free for 3 Days!