Home | Business News | Browse by Publication | I | International Journal of Market Research

Can online polls produce accurate findings?

Article Excerpt
This paper examines the relationship between traditional polling methods and recently developed internet polling methods. The validity of exercises to compare the two methods is discussed, the conclusion being that it is better to test polling figures against real world events with measurable outcomes. The challenges facing online polling companies in constructing valid samples and analysing responses are also examined.

Introduction

Ever since YouGov was founded in the spring of 2000 as an online polling and e-consultation company, it has been the subject of comment, inquiry, fascination and criticism. This is not surprising for, compared with traditional survey companies, its methods are radically different, its costs are lower and its media profile, relative to its size, far higher. If it is able to produce reliable data, then it is plainly in a strong, competitive position. Equally, its rivals would be able to defeat its challenge were they to prove that its methods do not work.

This paper is intended to inform the debate about internet polling. However, the right context must be set for that debate. Were the record of traditional companies perfect (or as perfect as the iron laws of sampling error permit), then the task of judging online surveys would be a simple one. An experiment could be designed in which an internet polling organisation and a traditional company posed the same questions at the same time to separate large samples employing their respective methods. The accuracy of online surveys could then be judged by how close their figures came to those obtained from face-to-face or telephone interviews.

Would that approach work? The recent record of traditional polling methods suggests not. In every election for more than a decade (including the Scottish Parliament elections in May 2003), they have displayed a persistent tendency to overstate Labour support and understate Conservative support. The one exception, ICM, has produced more accurate figures on two occasions; however, its methods, involving new ways of weighting and adjusting raw data, have been controversial. Even ICM, whose figures for the 1997 and 2001 general elections came close to the results of both contests, is not infallible: its poll for the BBC at the time of the 1999 European election failed to detect the significant Conservative victory (Independent 1999).

In short, there is neither perfection nor consensus in the world of traditional polling. This means that the reliability of online methods cannot properly be judged against the data produced by conventional methods. If a comparative exercise produced different figures, how could anyone tell which method (if either) produced more accurate results? A better approach is to test polling figures against real events with measurable outcomes. The next section reviews YouGov's record. The company has predicted the outcomes of six contests. In three cases it performed better than its rivals; in the other three cases YouGov was alone, and produced figures that came within one percentage point of the actual result. The third section discusses these issues. It examines the case for all survey research companies (other than those with the time and resources to conduct pure random surveys and obtain high response rates) seeking to 'model' public opinion from the raw data, instead of merely adding up the raw numbers, applying routine demographic weights and publishing the results. Naturally, any online polling company must be aware of the challenges it faces, in constructing valid samples and analysing the responses it receives. The fourth section describes YouGov's methods and its approach to modelling public opinion. The fifth section compares the results of an ICM experiment in online polling with the results from a parallel YouGov survey, and suggests some lessons to be learned.

YouGov's record

Between June 2001 and May 2003, YouGov polled six events with measurable outcomes. Table 1 shows its performance.

In the 2001 UK general election, YouGov came closer than any other company to the true size of Labour's lead (it was 9.3%; YouGov's poll predicted a 10% lead). It was the only company not to understate the level of Conservative support.

YouGov's prediction for the 2001 Australian general election was more accurate than that of any other company for the first vote, and matched the most accurate of the other companies in its prediction of the two-party division of support after the distribution of second preferences.

YouGov was the only company to conduct a nationwide survey of party members for the 2001 Conservative leadership election. Its survey, for the Daily Telegraph, exactly predicted the scale of Iain Duncan Smith's victory over Kenneth Clarke. Similarly, YouGov's survey just before the final of ITV's Pop Idol contest (February 2002) precisely predicted the scale of Will Young's victory over Gareth Gates, at a time when virtually all 'expert' opinion (and bookmakers) predicted victory for Gareth Gates.

YouGov was also alone in predicting the result of the May 2002 London borough elections (for the Evening Standard). Its figures for the three-party division of support all came within one percentage point of the result. (1)

With YouGov's poll for the Daily Telegraph for the May 2003 Scottish Parliament elections, its record can once again be compared with not only the result but also the polls of other companies. Four polls published figures in election week. YouGov's average error, at 1.7%, was less than any other company's. According to Professor John Curtice of Strathclyde University:

Election night is not just a test for the politicians. It is also a crucial examination for the media and the pollsters ... The pollsters have been dogged by a persistent problem ever since the 1992 UK election--a tendency to underestimate the Conservative vote and overestimate Labour's. The problem is still there. This time, too, all of the final polls seriously overestimated Labour's share of the first vote and underestimated the Conservatives' tally on both votes--except for one. The exception was the poll conducted by the internet company YouGov. It has already stolen a large proportion of the polling market from its more established rivals and its performance on Thursday will only enhance its reputation. (Scotsman 2003)

Learning from the record of traditional survey methods

1992 was the annus horribilis for traditional polling companies. In that year's general election, the Conservatives won the popular vote by a margin of eight percentage points; yet none of the election-day polls reflected this.

An inquiry set up by the Market Research Society pointed to a number of failings. Polling companies responded...

View this article FREE - Now for a Limited Time, try Goliath Business News
Free for 3 Days!