Home | Business News | Browse by Publication | J | Journal of the American Statistical Association

On priors with a Kullback-Leibler property.

Article Excerpt
1. INTRODUCTION.

Recent Bayesian nonparametric literature has focused on consistency properties of Bayesian procedures (see, e.g., Wasserman 1998; Barron, Schervish, and Wasserman 1999; Ghosal, Ghosh, and Ramamoorthi 1999). Based on the results of these authors and on our own results, we argue that it is recommendable that Bayesians use priors that put positive mass on all Kullback-Leibler neighborhoods of all densities. A Kullback-Leibler neighborhood of size [epsilon] > of the density g is given by

[A.sub.g]([epsilon]) = {f: [integral] g(x)log{g(x)/f(x)}dx < [epsilon]}.

The property of the prior [PI] with which we are concerned is that

[PI]{[A.sub.g]([epsilon])} >

for all [epsilon] > and all densities g. We call this the Kullback-Leibler property for [PI]. To achieve this, a nonparametric prior is required. (For specific examples of priors with the above foregoing property, see Barron et al. 1999; Ghosal et al. 1999; Petrone and Wasserman 2002.)

We consider only the case in which [f.sub.0] is a density function and [X.sup.n] = ([X.sub.1],..., [X.sub.n]) is available as an independent and identically distributed random sample from [f.sub.0], the first n observations of a possibly infinite sequence of reals [X.sub.1], [X.sub.2],.... Because [f.sub.0] is unknown, the Bayesian constructs a prior distribution on the relevant space of density functions, or distribution functions, reflecting available prior information about the location of [f.sub.0]. Assuming that all the densities under consideration are dominated by some [sigma]-finite measure, which we take to be the Lebesgue measure, Bayes's theorem and the data [X.sup.n] combine to update the prior to the posterior.

In this article we demonstrate that the Kullback-Leibler property for a prior [PI] provides good large sample properties for a number of Bayes procedures. Consequently, we argue that Bayesians should be constructing priors with the Kullback-Leibler property, at the very least when there is doubt about the underlying shape of the density function generating the data. The results are based on large samples, highly relevant these days when large datasets are becoming the norm. For example, financial datasets can be recorded in the tens of thousands.

In particular, from a Bayes factor perspective, we demonstrate that if a model has the Kullback-Leibler property, then the Bayes factor always supports this model eventually when compared with any other model without the Kullback-Leibler property. Practically speaking, a model with this property meets the requirements of an "asymptotically true" model.

We should point out that all Bayesian models [i.e., M = {f(x; 0), [pi](0)}] define a prior probability [PI] on the space of density functions. A random density function from [PI] is chosen by first choosing a [theta] from [pi] and then putting f(*) [equivalent to] f([??] [theta]). Hence for us, a Bayesian model is precisely the prior [PI] on the space of density functions. A parametric model of finite dimensions will not satisfy the Kullback-Leibler property, unless [f.sub.0] is known to belong to the particular parametric family.

The following reasons suggest that [PI] should have the Kullback-Leibler property:

1. Many practicing statisticians would argue that parametric models are sufficient when combined with model checking and model comparison diagnostics (see, e.g., Bernardo and Smith 1994). However, Draper (1999), in an insightful discussion of the article by Walker et al. (1999), pointed out that allocating probability mass 1 to parametric subsets of densities should not be done lightly. The reason for this is that switching models when the original model(s) under consideration is (are) found to be deficient in some sense exposes the statistician to the very real possibility of poor calibration. Therefore, there is a very practical reason for assigning mass 1 to the set of all densities; the data can offer no surprises. Some authors go even further, pointing out that the allocation of probability 1 to a parametric model and a desire to check this allocation once the data has arrived represents an internal contradiction (see Lindsey 1999).

2. If [PI] has the Kullback-Leibler property, then the Bayes factor comparing this model with any...