Quality reports for hospitals inconsistent

Leading sources for quality data rank the same facilities differently, even for the same conditions. Ratings providers say patients can judge for themselves.

By Kevin B. O’Reilly — Posted Jan. 12, 2009

Print  |   Email  |   Respond  |   Reprints  |   Like Facebook  |   Share Twitter  |   Tweet Linkedin

The promise of public quality reporting is undermined by competing sources of information that use different metrics, methodologies and data sources, yielding contradictory and confusing data for patients to sort through, says a study in the November/December 2008, Health Affairs.

Researchers compared how nine 250-plus-bed general hospitals in the Boston area fared on five leading hospital quality reporting services -- HealthGrades, Medicare's Hospital Compare, the Leapfrog Group, U.S. News & World Report's "America's Best Hospitals," and Massachusetts Healthcare Quality and Cost -- and found results were all over the map.

Hospitals that did well on one measure, say total hip replacement, might get a mediocre grade on community-acquired pneumonia from the same service, making it hard for patients to judge overall quality.

Even when looking at how the different reporting systems graded hospitals on the same condition or procedure, rankings varied significantly. For example, two hospitals that earned the top spot for coronary artery bypass graft surgery from at least one service were given fourth and last places by another service.

"I don't think these ratings are at the point where doctors or patients can really use them," said Michael B. Rothberg, MD, MPH, lead author of the study and assistant professor of medicine at Tufts University School of Medicine in Boston. "This is a great idea in concept but very hard to do in practice, and we need a lot more thought about how to do this before it's going to be useful."

Dr. Rothberg formerly served as associate medical director for quality at Baystate Medical Center in Springfield, Mass., where study co-author Evan M. Benjamin, MD, is vice president for quality. The physicians were frustrated by how "hospitals would appear good in one system but average in another," Dr. Benjamin said.

"Public reporting was created to be a tool for consumer choice and to improve quality, but the inconsistency in these systems does a disservice to patients rather than achieve its true potential," he said.

The quality reporting services defended their approaches.

Avery Comarow, editor of U.S. News' "America's Best Hospitals" supplements, said it should be no surprise that competing services yield different rankings. He said the study's authors seemed to put too little faith in patients' ability to make their own judgments.

"People may be confused by these ratings, like anything else that's complicated," Comarow said. "But people who are sufficiently motivated will at least attempt to penetrate the different approaches."

Scott Shapiro, a spokesman for HealthGrades, said his service and Hospital Compare are "the only ratings that the public accesses online with any frequency ... so there is not a lot of opportunity for confusion."

The Centers for Medicare & Medicaid Services did not respond to an AMNews inquiry by deadline.

Seeking harmony

Leapfrog Group CEO Leah Binder argued that her organization's rankings "are absolutely consistent with the best-known, peer-reviewed evidence in the field." Nevertheless, she agreed that more consistency is needed.

"We don't want to see hospitals spend all their time reporting to surveys instead of trying to improve their performance on the surveys," Binder said. "We stand ready to work with any hospitals to advocate for harmony in the measures."

The American Medical Association, in the context of policy addressing pay-for-performance programs, says that "accurate data and scientifically valid analytical methods" should be used to assess physician performance. The AMA-convened Physician Consortium for Performance Improvement has developed 200 quality improvement metrics for public reporting.

The American Hospital Assn., through its participation in the Hospital Quality Alliance, has long called for a consistent, evidence-based approach, said Nancy Foster, the organization's vice president for quality and patient safety.

"This [study] shows how vital it will be to accomplish this standardization," Foster said.

Back to top


Reports are little seen, less often used

Quality reports may be all over the map in their verdicts, but patients seem to be on the same page in paying little attention to them. Only three in 10 patients reported seeing quality information of any kind on health plans, hospitals or physicians, says a Kaiser Family Foundation survey of 1,517 adults, released in October 2008. That is down 17% from 2006 and has dropped 23% since 1996, the first year Kaiser surveyed patients on the matter. The proportion of patients who used the quality reports to make decisions is even lower.

Patients using reports
Type of report 1996 2000 2004 2006 2008
Health plans 12% 9% 13% 12% 9%
Hospitals 6% 4% 8% 10% 7%
Physicians 4% 4% 6% 7% 6%

Source: "2008 Update on Consumers' Views of Patient Safety and Quality Information," Henry J. Kaiser Family Foundation, October 2008 (link)

Back to top

External links

"Choosing the Best Hospital: The Limitations of Public Quality Reporting," abstract, Health Affairs, November/December 2008 (link)

"U.S. News's 'Best Hospitals' Clashes With Other Ratings. Is That Bad?" Comarow on Quality, Nov. 14, 2008 (link)

"2008 Update on Consumers' Views of Patient Safety and Quality Information," Henry J. Kaiser Family Foundation, October 2008 (link)

Back to top



Read story

Confronting bias against obese patients

Medical educators are starting to raise awareness about how weight-related stigma can impair patient-physician communication and the treatment of obesity. Read story

Read story


American Medical News is ceasing publication after 55 years of serving physicians by keeping them informed of their rapidly changing profession. Read story

Read story

Policing medical practice employees after work

Doctors can try to regulate staff actions outside the office, but they must watch what they try to stamp out and how they do it. Read story

Read story

Diabetes prevention: Set on a course for lifestyle change

The YMCA's evidence-based program is helping prediabetic patients eat right, get active and lose weight. Read story

Read story

Medicaid's muddled preventive care picture

The health system reform law promises no-cost coverage of a lengthy list of screenings and other prevention services, but some beneficiaries still might miss out. Read story

Read story

How to get tax breaks for your medical practice

Federal, state and local governments offer doctors incentives because practices are recognized as economic engines. But physicians must know how and where to find them. Read story

Read story

Advance pay ACOs: A down payment on Medicare's future

Accountable care organizations that pay doctors up-front bring practice improvements, but it's unclear yet if program actuaries will see a return on investment. Read story

Read story

Physician liability: Your team, your legal risk

When health care team members drop the ball, it's often doctors who end up in court. How can physicians improve such care and avoid risks? Read story

  • Stay informed
  • Twitter
  • Facebook
  • RSS
  • LinkedIn