Home | Business News | Browse by Publication | I | International Journal of Market Research

Assessment of survey data quality: a pragmatic approach focused on interviewer tasks.

Article Excerpt
Within the community of survey researchers there has been an increasing awareness that the total survey error approach has only partially realised its objective of setting up a model to estimate the total of all error components. Insufficient attention has been paid to non-sampling error. In the total quality management (TQM) approach the focus is on the production process. This is a comprehensive attempt to motivate everyone involved in the production of survey data to make permanent improvements to all components of the process. The TQM approach is suitable for organisations producing statistical data. This paper investigates the possibilities of integrating the major components of both approaches of data quality within the context of face-to-face interviews. The conceptual framework of a pragmatic approach is built on the concepts derived from evaluation research, such as process and product evaluation, and on the major tasks of the interviewer. The assessment of data quality should cover the process and output aspects of both the sample obtained and the registered responses. Within each stage of this assessment, a series of procedures has been identified. This is a very useful strategy that can be applied by organisations that attach great importance to the quality of their data. It is important to note that the proposed procedures for data quality assessment by themselves reduce errors.

Introduction

It makes sense to ask questions about the quality of data gathered for a particular research project. Answering these questions, however, is less obvious. After all, the quality of data can be assessed from very different angles (Platek & Sarndal 2001). Statisticians, for instance, will tend to focus on the sample design, the estimation procedures and the methods used for analysis. The construction of the questionnaire, the training of the interviewers, the assessment of interviewer behaviour as well as the response rate are important focal points for the survey methodologist. The researcher, for whom the substantive content is all-important, will be mainly concerned with the validity of the measurements. This diversity of potential focal points means that the quality of survey data is a complex, multi-dimensional concept that can only be described in general terms. Remarkable is the central role played by the concept of total survey error (TSE) for the way the quality of the data is defined. In fact, the quality of data is described as the relative absence of systematic and variable errors. In other words, quality is negatively defined. Recently, there has been an increasing awareness that the TSE approach has not sufficiently realised the objective of setting up a model to estimate the total of all error components. Insufficient attention has been paid to the non-sampling error (Biemer 2001; Platek & Sarndal 2001).

Quality of survey data can also be tackled from the angle of total quality management (TQM). TQM is a comprehensive approach motivating everyone involved in the production process to make permanent improvements to all components of the process. The underlying assumption is that all components of the production process individually and in correlation with each other contribute directly or indirectly to the quality of the end-product.

The continuous improvement in quality of products and services is subsequently considered to be a shared responsibility (Carton 1999). The TQM approach is suitable for organisations producing statistical data, for example the national statistical institutes and survey research organisations. It is at the forefront of the work of the Leadership Expert Group on Quality for the European Statistical System, which comprises Eurostat and the national statistical institutes associated with it. It emphasises that accuracy--measured by the mean squared error--is no longer the sole measure of data quality. In that sense, the TQM approach takes a different stand from the limited TSE approach. Compared to the latter, TQM of statistical data aims to include more factors relevant to the evaluation of data quality. Next to accuracy, the relevance, comparability, coherence and completeness come under scrutiny. TQM is a specifically organisational approach with strong emphasis on process quality as a prerequisite for quality of output, unlike the TSE approach, which is more project-specific, considering the output quality of the survey as its main objective. For TSE, quality is mainly expressed in terms of accuracy.

The question arises as to how particular aspects of both approaches of data quality can be integrated and implemented within the organisation and evaluation of one particular survey. It means investigating which ideas from TQM, in combination with the most important concepts from TSE, could be adapted to the level and context of a particular survey. The authors will limit themselves to surveys gathering data through face-to-face interviews. Furthermore, all activities relating to sampling and the construction of the questionnaire fall outside the scope of the investigation. The authors' starting-point is consequently the way a survey is conducted for which a suitable questionnaire and sample are available.

Below, a conceptual framework is presented integrating the main points of interest and basic concepts of TQM and TSE. It can be used for evaluating important aspects of the data quality of surveys. Furthermore, the authors will examine how the conceptual framework can be consolidated, and what the actual implications are for the organisation of surveys.

Conceptual framework for the assessment of survey data quality

For the execution of a survey, the interviewer is considered to be an important agent in the data production process. After all, he/she implements the complete survey design (Groves 1989). The interviewer's task consists of two important components. The first is to contact respondents, to obtain their cooperation and to conduct the actual interview in the narrow sense. The latter covers the questioning, the giving of instructions, probing for more information and the registration of the respondent's answers. Contacting respondents and obtaining their co-operation results in the realised or 'obtained' sample. The registered responses are, in the narrow sense, the output of the interview. The great emphasis placed by TQM on the optimal proceeding of the production process can be applied to a concrete survey situation by the way the interviewer is prepared for both components of the task, the way he/she completes the task and the feedback received in the process. TQM at project level consequently means that the process will be thoroughly evaluated. An important aspect is the extent to which interviewers follow prescribed rules and procedures while carrying out their task.

Quality assessment implies an evaluation of output as well as of the process. The central question in an output evaluation closely resembles...

View this article FREE - Now for a Limited Time, try Goliath Business News
Free for 3 Days!