|
Article Excerpt
Stimulated in large part by the publication in January 2000 of the Institute of Medicine report, To Err Is Human." Building a Safer Health System, the U.S. Congress established patient safety as a national priority in FY 2001 and gave AHRQ the mandate to lead federal patient safety improvement activities. In a multifaceted approach, AHRQ funded projects to develop new knowledge on patient safety epidemiology and practices, developed infrastructure components to strengthen support for patient safety improvements, and pursued dissemination activities to encourage adoption of effective practices.
To understand how the initiative was progressing, AHRQ contracted with RAND in September 2002 to serve as its Patient Safety Evaluation Center (evaluation center) and to perform a longitudinal, formative evaluation of the patient safety initiative. This 4-year evaluation, completed in September 2006, examined a broad range of components of the initiative. As described in the first article in this issue (Farley and Battles 2008), the evaluation design was based on the CIPP model, which encompasses the full spectrum of factors involved in the operation of a program (Stufflebeam et al. 1971; Stufflebeam, Madaus, and Kellaghan 2000), including the four evaluation components represented by the CIPP acronym: context, input, process, and product.
Collectively, the articles in this issue address the CIPP evaluation components that we performed in evaluating the patient safety initiative. The context and input evaluations, which are summarized in the introductory article (Farley and Battles 2008), examined the strategic aspects of the initiative. The process evaluation focused on its more operational aspects. Examples of how our process evaluation assessed specific activities of the initiative are presented in the second through sixth articles (Sorbero et al. 2008; Taylor et al. 2008; Damberg et al. 2008; Teleki et al. 2008; Mendel et al. 2008) We examined many more such activities in the evaluation, with results reported in four evaluation reports (Farley et al. 2005, 2007a, b, 2008). The seventh article (Greenberg et al. 2008) summarizes our product evaluation analyses, which was developmental work to help AHRQ prepare for monitoring impacts of the initiative on various stakeholders.
In this paper, we focus on process evaluation results, synthesizing our overall findings regarding AHRQ's progress in implementing its patient safety initiative over the 7-year period of 2000-2006. We also present suggested next steps for the initiative based on collective results from our context, input, process, and product evaluations.
The purpose of a process evaluation is to assess the evolution of activities undertaken in a program, successes or failures in executing those activities, and factors contributing to its progress. The process evaluation serves an important role in its own right by providing information about which aspects of the program are working well and what changes are needed to improve aspects that are not as successful. In addition, process evaluation results can be used to help interpret program outcomes, to better understand the dynamics of the activities that led to either positive or negative outcomes (Rossi and Freeman 1993; Stufflebeam, Madaus, and Kellaghan 2000).
A FRAMEWORK FOR THE PROCESS EVALUATION
To guide the process evaluation, we developed a conceptual framework that defines a five-component system for improving patient safety. These five components (which we call "system components") work together to achieve improved practices in the health care system. We assessed how well AHRQ's patient safety initiative had succeeded at contributing to each component:
Monitoring Progress and Maintaining Vigilance. Establishment and monitoring of measures to assess performance improvement progress for key patient safety processes or outcomes, while maintaining continued vigilance to ensure timely detection and response to issues that represent patient safety risks and hazards.
Knowledge of Epidemiology of Patient Safety Risks and Hazards. Identification of medical errors and causes of patient injury in health care delivery, with a focus on populations that are vulnerable because they are compromised in their ability to function as engaged patients during health care delivery.
Development of Effective Practices and Tools. Development and field testing of patient safety practices to identify those that are effective, appropriate, and feasible for health care organizations to implement, taking into account the level of evidence needed to assess patient safety practices.
Building Infrastructure for Effective Practices. Establishment of the health care structural and environmental elements needed for successful implementation of effective patient safety practices, including an organization's commitment and readiness to improve patient safety (e.g., culture, information systems), hazards to safety created by the organization's structure (e.g., physical configurations, procedural requirements), and effects of the macroenvironment on the organization's ability to act (e.g., legal and payment issues).
Achieving Broader Adoption of Effective Practices. The adoption, implementation, and institutionalization of improved patient safety practices to achieve sustainable improvement in patient safety performance across the health care system.
THREE-PART METHOD FOR ASSESSING OVERALL PROGRESS
The process evaluation was designed to assess the contributions of the patient safety initiative to strengthening each of the five system components in the framework presented above. The summary results developed from these analyses are qualitative. To obtain as rich an understanding as possible of its contributions, we used three separate analyses and compared their results in summary assessments. By drawing upon three sets of results, we could interpret findings with more confidence than would have been possible if we used only one set of analyses.
The first analysis (analysis 1) was an assessment of how well the initiative was performing in meeting the goals and targets that AHRQ had established for itself, relative to each system component. The second one (analysis 2) was our own comprehensive assessment of how the key activities of the patient safety initiative collectively contributed to each of the five system components. The third one (analysis 3) was assessment of ratings by national stakeholders involved in patient safety, for which we gathered data in individual interviews on their views regarding progress in improving patient safety in the United States, AHRQ's contribution to progress, and the need for future efforts to make further gains.
In preparing for analyses 1 and 2, the first data collection step was to identify the various activities AHRQ was pursuing in the patient safety initiative, through review of written documents, lists of funded projects, and interviews with AHRQ staff. Some of the key activities are addressed in five articles in this issue--funding research and development projects (Sorbero et al. 2008), patient safety implementation projects (Taylor et al. 2008), health information technology (health IT) projects (Damberg et al. 2008), Patient Safety Improvement Corps training program (Teleki et al. 2008), and partnership formation with other organizations (Mendel et al. 2008). Examples of other activities are development of the Patient Safety Indicators (PSI), patient safety evidence report, PSNet Web site, patient safety culture survey, and the TeamSTEPPS package. See Farley et al. (2008) for detailed listings of activities in the initiative.
We then collected and analyzed data for each key activity, through which we documented and assessed progress, successes, challenges, and effects. In most cases, we used a combination...
|
