Quality reports for hospitals inconsistent

The promise of public quality reporting is undermined by competing sources of information that use different metrics, methodologies and data sources, yielding contradictory and confusing data for patients to sort through, says a study in the November/December 2008, Health Affairs.

Researchers compared how nine 250-plus-bed general hospitals in the Boston area fared on five leading hospital quality reporting services -- HealthGrades, Medicare's Hospital Compare, the Leapfrog Group, U.S. News & World Report's "America's Best Hospitals," and Massachusetts Healthcare Quality and Cost -- and found results were all over the map.

Hospitals that did well on one measure, say total hip replacement, might get a mediocre grade on community-acquired pneumonia from the same service, making it hard for patients to judge overall quality.

Even when looking at how the different reporting systems graded hospitals on the same condition or procedure, rankings varied significantly. For example, two hospitals that earned the top spot for coronary artery bypass graft surgery from at least one service were given fourth and last places by another service.

"I don't think these ratings are at the point where doctors or patients can really use them," said Michael B. Rothberg, MD, MPH, lead author of the study and assistant professor of medicine at Tufts University School of Medicine in Boston. "This is a great idea in concept but very hard to do in practice, and we need a lot more thought about how to do this before it's going to be useful."

Dr. Rothberg formerly served as associate medical director for quality at Baystate Medical Center in Springfield, Mass., where study co-author Evan M. Benjamin, MD, is vice president for quality. The physicians were frustrated by how "hospitals would appear good in one system but average in another," Dr. Benjamin said.

"Public reporting was created to be a tool for consumer choice and to improve quality, but the inconsistency in these systems does a disservice to patients rather than achieve its true potential," he said.

The quality reporting services defended their approaches.

Avery Comarow, editor of U.S. News' "America's Best Hospitals" supplements, said it should be no surprise that competing services yield different rankings. He said the study's authors seemed to put too little faith in patients' ability to make their own judgments.

"People may be confused by these ratings, like anything else that's complicated," Comarow said. "But people who are sufficiently motivated will at least attempt to penetrate the different approaches."

Scott Shapiro, a spokesman for HealthGrades, said his service and Hospital Compare are "the only ratings that the public accesses online with any frequency ... so there is not a lot of opportunity for confusion."

The Centers for Medicare & Medicaid Services did not respond to an AMNews inquiry by deadline.

Seeking harmony

Leapfrog Group CEO Leah Binder argued that her organization's rankings "are absolutely consistent with the best-known, peer-reviewed evidence in the field." Nevertheless, she agreed that more consistency is needed.

"We don't want to see hospitals spend all their time reporting to surveys instead of trying to improve their performance on the surveys," Binder said. "We stand ready to work with any hospitals to advocate for harmony in the measures."

The American Medical Association, in the context of policy addressing pay-for-performance programs, says that "accurate data and scientifically valid analytical methods" should be used to assess physician performance. The AMA-convened Physician Consortium for Performance Improvement has developed 200 quality improvement metrics for public reporting.

The American Hospital Assn., through its participation in the Hospital Quality Alliance, has long called for a consistent, evidence-based approach, said Nancy Foster, the organization's vice president for quality and patient safety.

"This [study] shows how vital it will be to accomplish this standardization," Foster said.