|
Article Excerpt
Key words: adaptation; bias; climate change; decision making; endangered species; expert opinion; extinction; evaluation; evidence-based principles; expert judgment; forecasting methods; global warming; habitat loss; mathematical models; scientific method; sea ice.
History: This paper was refereed.
**********
Despite widespread agreement that the polar bear population increased during recent years following the imposition of stricter hunting rules (Prestrud and Stirling 1994), new concerns have been expressed that climate change will threaten the survival of some subpopulations in the 21st century. Such concerns led the US Fish and Wildlife Service to consider listing polar bears as a threatened species under the United States Endangered Species Act. To list a species that is currently in good health must surely require valid forecasts that its population would, if it were not listed, decline to levels that threaten the viability of the species. The decision to list polar bears thus rests on long-term forecasts.
The US Geological Survey commissioned nine administrative reports to satisfy the request of the Secretary of the Interior and the Fish and Wildlife Service to conduct analyses. Our objective was to determine if the forecasts were derived from accepted scientific procedures. We first examined the references in the nine government reports. We then assessed the forecasting procedures described in two of the reports relative to forecasting principles. The forecasting principles that we used are derived from evidence obtained from scientific research that has shown the methods that provide the most accurate forecasts for a given situation and the methods to avoid.
Scientific Forecasting Procedures
Scientists have studied forecasting since the 1930s; Armstrong (1978, 1985) provide summaries of important findings from the extensive forecasting literature.
In the mid-1990s, Scott Armstrong established the Forecasting Principles Project to summarize all useful knowledge about forecasting. The evidence was codified as principles, or condition-action statements, to provide guidance on which methods to use under different circumstances. The project led to the Principles of Forecasting handbook (Armstrong 2001). Forty internationally recognized forecasting-method experts formulated the principles and 123 reviewed them. We refer to the evidence-based methods as scientific forecasting procedures.
The strongest evidence is derived from empirical studies that compare the performance of alternative methods; the weakest is based on received wisdom about proper procedures. Ideally, performance is assessed by the ability of the selected method to provide useful ex ante forecasts. However, some of the principles seem self-evident (e.g., "provide complete, simple, and clear explanations of methods") and, as long as they were unchallenged by the available evidence, were included in the principles list.
The principles were derived from many fields, including demography, economics, engineering, finance, management, medicine, psychology, politics, and weather; this ensured that they encapsulated all relevant evidence and would apply to all types of forecasting problems. Some reviewers of our research have suggested that the principles do not apply to the physical sciences. When we asked them for evidence to support that assertion, we did not receive useful responses. Readers can examine the principles and form their own judgments on this issue. For example, does the principle, "Ensure that information is reliable and that measurement error is low," not apply when forecasting polar bear numbers?
The forecasting principles are available at www.forecastingprinciples.com, a website that the International Institute of Forecasters sponsors. The directors of the site claim that it provides "all useful knowledge about forecasting" and invite visitors to submit any missing evidence. The website also provides forecasting audit software that includes a summary of the principles (which currently number 140) and the strength of evidence for each principle; Armstrong (2001) and papers posted on the website provide details.
General Assessment of Long-Term Polar Bear Population Forecasts
We examined all references cited in the nine US Geological Survey Administrative Reports posted on the Internet. The reports, which included 444 unique references, were Amstrup et al. (2007), Bergen et al. (2007), DeWeaver (2007), Durner et al. (2007), Hunter et al. (2007), Obbard et al. (2007), Regehr et al. (2007), Rode et al. (2007), and Stirling et al. (2007). We were unable to find references to evidence that the forecasting methods described in the reports had been validated.
Forecasting Audit of Key Reports Prepared to Support the Listing of Polar Bears
We audited the forecasting procedures in the reports that we judged provided the strongest support (i.e., forecasts) for listing polar bears. We selected Amstrup et al. (2007), which we will refer to as AMD, because the press had discussed their forecast widely. We selected Hunter et al. (2007), which we will refer to as H6, because the authors used a substantially different approach to the one reported in AMD.
The reports provide forecasts of polar-bear populations for 45, 75, and 100 years from the year 2000 and make recommendations with respect to the polar-bear-listing decision. However, their recommendations do not follow logically from their research because they only make forecasts of the polar bear population. To make policy recommendations based on forecasts, the following assumptions are necessary:
(1) Global warming will occur and will reduce the amount of summer sea ice;
(2) Polar bears will not adapt; thus, they will obtain less food than they do now by hunting from the sea-ice platform;
(3) Listing polar bears as a threatened or endangered species will result in policies that will solve the problem without serious detrimental effects; and
(4) Other policies would be inferior to those that depend on an Endangered Species Act listing.
Regarding the first assumption, both AMD and H6 assumed that general circulation models (GCMs) provide scientifically valid forecasts of global temperature and the extent and thickness of sea ice. AMD stated: "Our future forecasts are based largely on information derived from general circulation model (GCM) projections of the extent and spatiotemporal distribution of sea ice" (AMD: p. 2; p. 83, Figure 2). H6 stated, "We extracted forecasts of the availability of sea ice for polar bears in the Southern Beaufort Sea region, using monthly forecasts of sea-ice concentrations from 10 IPCC Fourth Assessment Report (AR4) fully-coupled general circulation models" (p. 11). (Note: IPCC is the Intergovernmental Panel on Climate Change.) That is, the forecasts of both AMD and H6 are conditional on long-term global warming leading to a dramatic reduction in Arctic sea ice during melt-back periods in spring, late summer, and fall.
Green and Armstrong (2007) examined long-term climate-forecasting efforts and were unable to find a single forecast of global warming that was based on scientific methods. When they audited the GCM climate modelers' procedures, they found that only 13 percent of the relevant forecasting principles were followed properly; some contraventions of principles were critical. Their findings were consistent with earlier cautions. For example, Soon et al. (2001) found that the current generation of GCMs is unable to meaningfully calculate the effects that additional atmospheric carbon dioxide has on the climate. This is because of the uncertainty about the past and present climate and ignorance about relevant weather and climate processes. Some climate modelers state that the GCMs do not provide forecasts. According to one of the lead authors of the IPCC's AR4 (Trenberth 2007),
... there are no predictions by IPCC at all. And there
never have been. The IPCC instead proffers "what
if" projections of future climate that correspond to
certain emissions scenarios. There are a number of
assumptions that go into these emissions scenarios.
They are intended to cover a range of possible self consistent
"story lines" that then provide decision makers
with information about which paths might be more
desirable.
AMD and H6 provided no scientific evidence to support their assumptions about any of the four issues that we identified above. Thus, their forecasts are of no value to decision makers. Nevertheless, we audited their polar-bear-population forecasting procedures to assess if they would have produced valid forecasts if the underlying assumptions had been valid.
In conducting our audits, we read AMD and H6 and independently rated the forecasting procedures described in the reports by using the forecasting audit software mentioned above. The rating scale ranged from -2 to +2; the former indicated that the procedures contravene the principle; the latter signified that it is properly applied. Following the initial round of ratings, we examined differences in our ratings to reach consensus. When we had difficulty in reaching consensus, we moved ratings toward "0." Principle 1.3 (Make sure forecasts are independent of politics) is an example of a principle that was contravened in both reports (indeed, in all nine). By politics, we mean any type of organizational bias or pressure. It is not unusual for different stakeholders to prefer particular forecasts; however, if forecasters are influenced by such considerations, forecast accuracy could suffer. The header on the title page of each of the nine reports suggests how the authors interpreted their task: "USGS Science Strategy to Support US Fish and Wildlife Service Polar Bear Listing Decision." A more neutral statement of purpose might have read "Forecasts of the polar bear population under alternative policy regimes."
While it was easy to code the two reports' procedures against Principle 1.3, the ratings were subjective for many principles. Despite the subjectivity, our ratings after the first round of analyses for each report were substantially in agreement. Furthermore, we readily achieved consensus by the third round.
The two reports did not provide sufficient detail to allow us to rate some of the relevant principles. As a result, we contacted the report authors for additional information. We also asked them to review the ratings that we had made and to provide comments. In their replies, the report authors refused to provide any responses to our requests. (See #2 in the Author Comments section at the end of this paper.)
In December 2007, we sent a draft of this article to all authors whose works we cited substantively and asked them to inform us if we had misinterpreted their findings. None objected to our interpretations. We also invited each author to review our paper but received no reviews from our requests.
Audit Findings for AMD
In auditing AMD's forecasting procedures, we first agreed that 24 of the 140 forecasting principles were irrelevant to the forecasting problem they were trying to address. We then examined principles for which our ratings differed. The process involved three rounds of consultation; after two rounds, we were able to reach consensus on ratings against all 116 relevant principles. We were unable to rate AMD's procedures against 26 relevant principles (Table A.3) because the paper lacked the necessary information. Tables A.1, A.2, A.3, and A.4 provide full disclosure of our AMD ratings.
Overall, we found that AMD definitely contravened 41 principles and apparently contravened an additional 32 principles. The authors provided no justifications for the contraventions. Of the 116 relevant principles, we could find evidence that AMD properly applied only 17 (14.7 percent) (Table A.4).
In the remainder of this section, we will describe some of the more serious problems with the AMD forecasting procedures by listing a selected principle and then explaining how AMD addressed it.
Principle 6.7: Match the forecasting method(s) to the situation.
The AMD forecasts rely on the opinions of a single polar bear expert. The report authors transformed these opinions into a complex set of formulae without using evidence-based forecasting principles. In effect, the formulae were no more than a codification of the expert's unaided judgments, which are not appropriate for forecasting in this situation.
One of the most counterintuitive findings in forecasting is that judgmental forecasts by experts who ignore accepted forecasting principles have little value in complex and uncertain situations (Armstrong 1978, pp. 91-96; Tetlock 2005). This finding applies whether the opinions are expressed in words, spreadsheets, or mathematical models. In relation to the latter, Pilkey and Pilkey-Jarvis (2007) provide examples of the failure of domain experts' mathematical models when they are applied to diverse natural science problems including fish stocks, beach engineering, and invasive plants. This finding also applies regardless of the amount and quality of information that the experts use because of the following:
(1) Complexity: People cannot assess complex relationships through unaided observations.
(2) Coincidence: People confuse correlation with causation.
(3) Feedback: People making judgmental predictions typically do not receive unambiguous feedback that they can use to improve their forecasting.
(4) Bias: People have difficulty in obtaining or using evidence that contradicts their initial beliefs. This problem is especially serious among people who view themselves as experts.
Despite the lack of validity of expert unaided forecasts, many public-policy decisions are based on such forecasts. Research on persuasion has shown that people have substantial faith in the value of such forecasts and that faith increases when experts agree with one another. Although they may seem convincing at the time, expert forecasts can, a few years later, serve as important cautionary tales. Cerf and Navasky's (1998) book contains 310 pages of examples of false expert forecasts, such as the Fermi award-winning scientist John von Neumann's 1956 prediction that "A few decades hence, energy may be free." Examples of expert climate forecasts that turned out to be wrong are easy to find, such as UC Davis ecologist Kenneth Wart's prediction during an Earth Day speech at Swarthmore College (April 22, 1970) that "If present trends continue, the world will be about four degrees colder in 1990, but eleven degrees colder in the year 2000. This is about twice what it would take...
|
