Study Finds Mixed Results on Effectiveness of Surgical Care Improvement Measures

An analysis of data on adherence to surgical care improvement measures finds that when analyzed as a composite infection-prevention score, the improvement measures were associated with a lower probability of post-operative infection. However, adherence to individual measures – the format of publicly reported performance data – was not associated with a significantly lower risk of infection, according to a study in the June 23/30 issue of JAMA.

“The Surgical Care Improvement Project (SCIP), a national quality partnership dedicated to reducing the rate of surgical complications, has developed 20 measures covering various discrete elements of patient care. There are nine publicly reported SCIP measures, six of which focus on postoperative infection prevention,” the authors write. Validated data from hospitals are reported on the Health & Human Services (HHS) Web site, Hospital Compare. “Despite broad support from national stakeholders and a significant investment of time and money by the hospitals to collect these data, no large-scale investigation has been undertaken to evaluate their effectiveness for improving outcomes in routine clinical care.”

Jonah J. Stulberg, MD, PhD, MPH, of Case Western Reserve University, Cleveland, and colleagues conducted a study to evaluate the association between the 6 infection-prevention SCIP measures and postoperative infection rates in a representative sample of hospital discharges in the United States. The study included data for discharges between July 2006 and March 2008 of 405,720 patients (69 percent white and 11 percent black; 46 percent Medicare patients; and 68 percent elective surgical cases) from 398 hospitals in the United States for whom SCIP performance was recorded and submitted for public report on the Hospital Compare Web site.

Three original infection-prevention measures (S-INF-Core) and all six infection-prevention measures (S-INF) were aggregated into two separate all-or-none composite scores. The measures are on processes that include prophylactic antibiotic administration, glucose control, hair removal from the surgical site, and maintaining normal body temperature during surgery.

There were 3,996 documented post-operative infections. The researchers found that demonstrated adherence to SCIP as measured through the all-or-none global composite S-INF was associated with a decreased likelihood of developing a post-operative infection from 14.2 to 6.8 postoperative infections per 1,000 discharges. “However, the S-INF-Core composite was not statistically significant in its association with decreased likelihood of developing a post-operative infection (from 11.5 to 5.3 postoperative infections per 1,000 discharges). In addition, reported adherence on individual SCIP items was not associated with decreased risk of postoperative infection,” the authors write.

“Based on our findings, the individual item performance rates reported publicly do not fulfill their stated purpose of pointing consumers toward high-quality hospitals. However, when taken in aggregate, improved performance on our global all-or-none composite measure is associated with improved outcomes at the discharge level. Therefore, while the individual items may not imply quality differences, the overall ability to demonstrate adherence to multiple SCIP processes of care may. Improved methods for identification of quality of care are necessary to be able to define improvements in patient outcomes, and to justify the massive investment of time and money in tracking these processes of care.”

Mary T. Hawn, MD, MPH, of the University of Alabama, Birmingham, writes in an accompanying editorial, “Surgical Care Improvement: Should Performance Measures Have Performance Measures,” that it appears that investing resources in SCIP reporting is no longer cost-effective. “Ideal surgical quality-improvement efforts would measure whether the right patient receives the right operation at the right time and whether the operation is effective. This is more challenging to measure given the complex nature of the health care system, including barriers to access, financial incentives, and the uniqueness of the therapeutic intent of individual surgical procedures. Current mandated surgical quality-improvement processes such as SCIP focus on incremental and narrow process measures that are purported to measure the overall quality of an episode of surgical care. Despite enormous resources committed to these measures and marked improvement in adherence, the evidence to date suggests that SCIP has not improved surgical outcomes. Future quality-improvement endeavors should have linkage between discrete performance and outcome measures so the effectiveness of combined efforts can be unequivocally measured and clearly evaluated.”

References: JAMA. 2010;303[24]:2479-2485; and JAMA. 2010;303[24]:2527-2528.