|
Article Excerpt
Following a successful idea generation exercise, a company might easily be left with hundreds of ideas generated by experts, employees, or consumers. The next step is to screen these ideas and identify those with the highest potential. In this paper we propose a practical approach to involving consumers in idea screening.
Although the number of ideas may potentially be very large, it would be unreasonable to ask each consumer to evaluate more than a few ideas. This raises the challenge of efficiently selecting the ideas to be evaluated by each consumer. We describe several idea-screening algorithms that perform this selection adaptively based on the evaluations made by previous consumers. We use simulations to compare and analyze the performance of the algorithms as well as to understand their behavior. The best-performing algorithm focuses on the ideas that are the most likely to have been misclassified (as "top" or "bottom" ideas) based on the previous evaluations, and avoids discarding ideas too fast by adding random perturbations to the misclassification probabilities. We demonstrate the convergent validity of this algorithm using a field experiment, which also confirms the convergence pattern predicted by simulations.
Key words: innovation; marketing research; marketing surveys; marketing tools; new product research; product development
1. Introduction
Idea generation is critical to new product development. It belongs to the "fuzzy front end" of the development process, recognized as a key leverage point for a firm (Dahan and Hauser 2001a, Hauser et al. 2006). A variety of idea generation methods have been introduced since the 1950s. The most popular is probably brainstorming (Osborn 1957). Other traditional examples include lateral thinking (De Bono 1970), synectics (Prince 1970, Gordon 1969), and six thinking hats (De Bono 1985). Recent developments include electronic brainstorming (Nunamaker et al. 1987; Gallupe et al. 1991, 1992; Dennis and Valacich 1993; Valacich et al. 1994), ideation templates (Goldenberg et al. 1999a, b; Goldenberg and Mazursky 2002), and incentives-based idea generation (Toubia 2006). With the development of Internet-based tools, companies are increasingly involving their own consumers in idea generation (Forbes 2005).
Depending on the method used, a successful idea generation exercise may result in up to hundreds of ideas generated by experts, consumers, or employees. The number of ideas appears even more likely to be large when consumers are involved in the process. (1) The new product development team is then left with the daunting task of screening these ideas in order to focus its limited resources on those with the highest potential. The selected ideas will be refined and translated into specific features or integrated products. In our field experiment, examples of consumer-generated ideas on how to improve cellular phones (see Table 2) included "There would be a way to ftp data files," "Download movies with your phone and project them on the wall so it seems like you're at the theater," etc. One traditional approach to idea screening is to ask one or a few experts to go over the transcripts of ideas and evaluate them (Urban and Hauser 1993). However, experts' judgments might not always reflect consumers' needs and preferences. (2)
In this paper we propose a practical approach to involving consumers in idea screening. Although the number of ideas to be screened may potentially be very large (especially if a large number of consumers have been involved in the idea generation process), it would be unreasonable to ask each consumer to evaluate more than a few ideas, especially if the evaluations are to be performed online in a noncontrolled environment (Dahan and Hauser 2001b). This raises the challenge of efficiently selecting the ideas to be evaluated by each consumer in order to converge to the best ideas as quickly (i.e., with only few respondents) and reliably as possible. We assume that the evaluations are done online and sequentially, allowing the selection to be performed adaptively based on the evaluations made by previous consumers.
We propose and explore several algorithms for adaptive idea screening. We assume that each idea appeals to an unknown proportion of consumers. Our estimate of this proportion follows a beta distribution with parameters depending on the previous evaluations. We assume that the team's objective is to identify the top m ideas out of a given set. We use simulations to compare and analyze the performance of the algorithms, as well as to understand their behavior and the drivers of differences in performance. We demonstrate the convergent validity of the best-performing algorithm using a field experiment, which also confirms the convergence pattern predicted by simulations. Note that our field experiment focuses on convergence and convergent validity, and that we rely on simulations to compare the performance of the different algorithms.
A problem with some similarities to ours was studied in the educational testing literature by Bradlow and Wainer (1998). Bradlow and Wainer consider subjective tests (e.g., essays) raters by human judges (on a continuous scale), resulting in binary pass/fail decisions (such that only candidates with an average grade above a predefined cutoff pass). They consider a situation in which the rescoring of some tests is possible after all tests have been rated by a fixed number of judges and initial pass/fail decisions have been made, and study the problem of allocating judges in the rescoring phase (e.g., which essays should be graded again). They find, using a modeling setup different from ours, (3) that for tests in which the number of initial failers and passers are approximately equal, a reasonable strategy is to rescore only examinees near the cutoff score (they compare this strategy to one where only failures are rescored).
Beyond the differences in modeling approach, context, and type of evaluations, two fundamental differences between our problem and the one studied by Bradlow and Wainer are that (1) the number of previous evaluations per item is constant across items in the latter (same initial number of raters on each essay) and different in the former (different number of previous evaluations per idea) and (2) allocation decisions are made once in the latter versus many times (once for each consumer) in the former. Given these differences, Bradlow and Wainer's work is not directly applicable to our problem. However, we will use it as an initial building block for some of our algorithms.
This paper is structured as follows. We introduce the idea selection algorithms in [section] 2. In [section] 3 we report the results of a series of simulations designed to study the performance and behavior of these algorithms. We report the results of our field experiment in [section] 4. We describe a managerial application of our research in [section] 5 and conclude in [section] 6.
2. Algorithms for Adaptive Idea Selection
Notations and Definitions
As mentioned earlier, we assume that our goal is to select a fixed number of ideas to be brought to the next stage of the new product development process, i.e., to identify the top m ideas out of a set of I previously generated ideas. In order to achieve this goal, we ask different consumers to evaluate different subsets of k ideas. For simplicity, we assume that consumers provide binary evaluations of the ideas, i.e., they indicate which ideas they believe to be "good." Note that we do not restrict the definition of a "good" idea. It can be specified by the researcher and should be explicitly given to the consumers before they start their evaluations. Note also that we show in Appendix A how our framework could be extended to nonbinary evaluations. We leave to future research the extension to other screening goals, such as identifying all ideas above a predefined threshold.
Let us define [([p.sub.i]).sub.i [member of]{1, ..., I}] as the probability that a randomly selected consumer will classify idea i as a good idea. We use [p.sub.i] as a measure of the quality of the idea, i.e., our goal is to identify the m ideas with the highest probabilities.
Let us define the following:
[([n.sub.Si]).sub.i [member of]{1, ..., I}] = number of respondents who have evaluated idea i and classified it as a good idea. (S stands for "success.")
[([n.sub.Fi]).sub.i [member of]{1, ..., I}] = number of respondents who have evaluated idea i and did not classify it as a good idea. (F stands for "failure.")
[n.sub.S0], [n.sub.F0]: parameters of our prior on [p.sub.i], assumed to follow a beta distribution Beta([n.sub.S0], [n.sub.F0]).
[([[??].sub.i]).sub.i [member of]{1, ..., I}] = our estimate of [p.sub.i], based on the previous evaluations. Given our beta prior and the fact that the evaluations follow a binomial likelihood, the posterior on [p.sub.i] follows another beta distribution: Beta([n.sub.S0] + [n.sub.Si], [n.sub.F0] + [n.sub.Fi]). (4) Our point estimate of [p.sub.i] is simply the expected value of this distribution: [p.sub.i] = ([n.sub.Si] + [n.sub.S0)/([n.sub.Fi] + [n.sub.Si] + [n.sub.S0] +...
|
