|
...models). The inherent variability the simulation's output arising from the dependence of the output on the simulation's random input streams is called stochastic uncertainty (Helton, 1997). Often we assume that the simulation's input models are known parametric families of probability distributions. However, in selecting a single input model from a list of alternative distribution families that could be used to represent a given random input stream, we usually lack complete knowledge, and thus have some degree of uncertainty, about how to make an appropriate selection. This second source of output variability is called model uncertainty (Raftery et al., 1996). The parameters on which the input models depend are often assumed to be unknown constants. In practice, input-model parameters are estimated from subjective information (expert opinion) or sample data observed on the input processes driving the real system (or both). The estimation of unknown input-model parameters gives rise to another source of output variability called parameter uncertainty (Raftery et al., 1996).
Cheng and Holland (1997) consider two methods for evaluating how the variance of the simulation output depends on parameter and stochastic uncertainties. The first method is based on classical differential analysis, or the delta method (Stuart and Ord, 1994). The main result of Cheng and Holland (1997) is that under some generally applicable regularity conditions, the variance of the simulation output is composed of two distinct terms depending respectively on parameter uncertainty and stochastic uncertainty. One problem with the Cheng-Holland delta method is that certain sensitivity coefficients must be estimated, and the computational effort needed to do this increases linearly with the number of input-model parameters. Moreover, when the number of input-model parameters is large, a problem can occur with spurious variability overinflating the estimate of the variance component due to parameter uncertainty. Cheng and Holland (1997) also examine the parametric bootstrap (Efron and Tibshirani, 1993) as a method for estimating the variance of the simulation output. Although the parametric bootstrap can be computationally more expensive than the delta method in some applications, it avoids the variance-estimation problems of the delta method. If the number of input-model parameters is large, then the parametric bootstrap can require substantially less computational effort than the delta method.
Both the delta method and the parametric bootstrap are based on the assumption that the input-model parameters are unknown but constant (deterministic) quantities. Moreover, the output inferences are implicitly conditional on the single selected input model. The objective of the simulation experiment is therefore to estimate the mean output response as a function of the "true" but unknown values of the input-model parameters. The fundamental problem with such approaches to simulation input modeling is that conditional on a single input model and on given values of the parameters of that input model, the output inference underestimates the overall (unconditional) uncertainty in the output quantities of interest, sometimes to a dramatic extent (Kass and Raftery, 1995). Moreover, the usual approach to model selection in the simulation community is commonly guided by a series of goodness-of-fit tests (Law and Kelton, 2000). These tests can be highly misleading and very difficult to interpret in a classical statistical framework (Berger and Delampady, 1987).
All the abovementioned difficulties can be avoided if we adopt a Bayesian approach that incorporates prior information on competing input models and their parameters in a rigorous manner. Using Bayes' rule, we can compute the posterior probability distributions for all competing input models and their parameters; and then we can make a composite inference that takes account of model and parameter uncertainty in a formally justifiable way (Zouaoui and Wilson, 2001a, 2001b). Even if prior information is not readily available, there are methods to perform a full Bayesian analysis that rely on noninformative priors and thus will give more weight to the observed data, but will still incorporate the model and parameter uncertainties that are due to our lack of knowledge about the simulation's input processes. These methods generalize the classical inferences conditional on the selection of a single input model and its parameters, and they work for both small and large samples.
In this paper, we use a Bayesian approach to account for both stochastic uncertainty and parameter uncertainty in simulation experiments; and we estimate the effects of these sources of uncertainty on the output quantity of interest. We assume that the functional forms of all input models are known, perhaps based on our prior knowledge of the processes driving the real system, a situation that sometimes occurs in certain types of applications. For example, in a reliability study of independent components with constant failure rates, each component's time-to-failure is randomly sampled from an exponential distribution whose mean is the reciprocal of the associated failure rate. Moreover, in many queueing systems the stream of customers arriving at any particular workstation from outside the system is characterized by customer interarrival times that are independent and identically distributed (i.i.d.) exponential random variables whose mean is the reciprocal of the associated arrival rate; see Section 5.4 below. Another reason for fixing all input models is to avoid excessive demands not only on the user's time and effort but also on the user's computing facilities in handling more than one input model for each random input process, especially in large-scale simulation experiments involving hundreds or thousands of probabilistic input processes. Moreover, in some situations the assumption of no model uncertainty facilitates the design and execution of Monte Carlo experiments that compare the performance of Bayesian and frequentist methods for simulation input modeling.
The rest of this paper is organized as follows. In Section 2 we define the notation used to describe our setup for probabilistic simulation experiments. In Section 3 we survey some recent developments concerning the use of the delta method and the parametric bootstrap for frequentist simulation input modeling. In Section 4 we detail our Bayesian framework for handling parameter and stochastic uncertainties in discrete-event simulation; and this development leads to our "Bayesian simulation replication algorithm" for designing simulation experiments. To evaluate the performance of both Bayesian and frequentist input-modeling techniques on a fair and consistent basis, in Section 5 we formulate appropriate evaluation criteria and illustrate the application of these criteria to simulations of a single-server queue and a communications network. This paper is based on Zouaoui (2001), with some of our initial results being first presented in Zouaoui and Wilson (2001c). We extend our basic approach to account for model uncertainty as well as parameter uncertainty and stochastic uncertainty in Zouaoui and Wilson (2001a, 2001b).
2. Setup for probabilistic simulation experiments
A probabilistic simulation experiment consists of making m independent runs of a simulation of a real system and observing y, a single output performance measure of primary interest, on each run. For simplicity, we assume that the real system is driven by a single sequence {[X.sub.1], [X.sub.2], ...} of i.i.d, input random variables, from which we observe the random sample x = ([x.sub.1], ..., [x.sub.n]) of size n. The setup described below is easily extended to handle multiple streams of random inputs to the real and simulated systems.
During the jth simulation run (j = 1, ..., m), a stream of random numbers [u.sub.j] = ([u.sub.j1], [u.sub.j2], ...) is generated internally within the simulation so that the components of [u.sub.j] are randomly sampled from the uniform distribution on the unit interval [0, 1 ]. The random-number stream [u.sub.j] is used to generate the simulation input stream [[??].sub.j] = ([[??].sub.j1], [[??].sub.j2], ...) by some transformation method, from which the output [y.sub.j] is finally computed. It is convenient to think of both streams [u.sub.j] and [[??].sub.j] as finite for a fixed simulation run length.
The inverse transtform method is commonly used in simulation software to obtain the input stream [[??].sub.j] from the random-number stream [u.sub.j]. If [u.sub.ji] denotes the ith random number generated on the jth simulation run and if [[??].sub.ji] denotes the ith input random variable sampled on the jth run, then [[??].sub.ji] is generated via the inverse transform method as follows
(1) [[??].sub.ji] = [G.sup.-1]([u.sub.ji], [theta),
where G(x, [theta) [equivalent to] Pr{[X.sub.i] [less than or equal to] x | [theta]} (for all x) is the marginal cumulative distribution function (c.d.f.) of the original input process {[X.sub.i]} with inverse c.d.f. [G.sup.-1](u, [theta]) [equivalent to] min {x: G(x, [theta]) [greater than or equal to] u} (for u [member of] [0, 1]), and where [theta] is the d-dimensional vector of unknown input-model parameters. Since we assume in this paper that the functional form...
NOTE: All illustrations and photos
have been removed from this article.
More articles from IIE Transactions
Robust optimization of experimentally derived objective functions., September 01, 2003
Designing an assembly process with stochastic material arrivals., September 01, 2003
Price competition of make-to-order firms., September 01, 2003
Market design and human trading behavior in electricity markets., September 01, 2003
Enhanced simulation inference using bootstraps of historical inputs., September 01, 2003
Looking for additional articles?
Search our database of over 3 million articles.
Looking for more in-depth information on this industry?
Search our complete database of Industry & Market reports by text, subject, publication
name or publication date.
About Goliath
Whether you're looking for sales prospects, competitive information, company
analysis or best practices in managing your organization,
Goliath can help you meet your business needs.
Our extensive business information databases empower business
professionals with both the breadth and depth of credible,
authoritative information they need to support their business
goals. Whether it be strategic planning, sales prospecting,
company research or defining management best practices -
Goliath is your leading source for accurate information.
|
