Despite warning, some hospitals get harsh patient safety grade

When the employer-backed Leapfrog Group handed out grades of A, B or C on patient safety to individual hospitals in June, 5% of hospitals scored low enough to receive D's or F's but were marked as “grade pending.” The hospitals had better shape up by the end of the year, Leapfrog said, or they will get a D or F.

Now the group has kept its promise, handing out the low marks to 146 U.S. hospitals, including highly regarded facilities such as Cleveland Clinic and the Ronald Reagan UCLA Medical Center in Los Angeles.

The criticism from hospital officials that greeted the initial round of Leapfrog grades has escalated now that D's and F's are on the report card. But Leapfrog's leaders are standing firm behind their scorecard's methodology, arguing that the grades are needed to help patients decide where to seek care and to spur dramatic hospital improvement on patient safety.

The Leapfrog grades, released Nov. 28, are based on 26 measures of patient safety and drawn from publicly available Medicare data for 2,618 general acute care hospitals. When applicable, grades also were based on hospitals' answers to the organization's patient safety survey or an American Hospital Assn. survey. Leapfrog's stated goal is to use their employer members' health purchasing power to influence quality and affordability.

Metrics cover areas such as intensive care unit physician staffing, medication reconciliation, use of computerized physician order entry, hand hygiene, antibiotic administration, catheter removal and a few “never events,” including air embolisms and serious pressure ulcers. A bad grade does not necessarily reflect the overall quality of care at a hospital, but does mean it is performing poorly on these safety metrics, Leapfrog officials said.

Hospitals without enough information available on at least 14 of the 26 measures do not receive a score and are not included in the report card. The formulation of the grading process was conducted by a nine-member panel that includes some of the most prominent names in the world of patient safety research, such as Harvard's Lucian Leape, MD.

A majority of hospitals got A's or B's, but 38% were given C's, nearly 5% got a D and fewer than 1% received a failing grade. The main Ohio campus of Cleveland Clinic — praised by President Obama during an October presidential debate as “one of the best health care systems in the world” and a model for how health care should be delivered — received a D. The Ronald Reagan UCLA Medical Center also has a sterling reputation, yet got an F from Leapfrog.

Hospitals question methodology

Hospital officials said Leapfrog's use of publicly available performance data in grading is problematic, because they capture only a slice of the care happening in hospitals, with grades derived from information collected as long ago as July 2009. Many also claim that the nature of the grading system gives preferential treatment to hospitals that answered Leapfrog's patient safety survey, compared with those that answered similar care questions on the AHA survey.

“Cleveland Clinic, like an increasing number of hospitals across the country, has declined to participate in the Leapfrog survey for the past two years,” the health system said in a statement. “We have not found this survey useful in prioritizing our patient safety efforts. Due to its methodology and limitations associated with the use of publicly available data, the Leapfrog report does not accurately reflect Cleveland Clinic's safety program or the many safety improvements throughout our system in recent years.”

In a statement, the Ronald Reagan UCLA Medical Center raised similar concerns, adding: “The problem with nearly all of the hospital report cards promoted by various organizations is that there is no consistency. The same hospital can rank high on one report card and low on another report card.”

Nancy Foster, the AHA's vice president of quality and patient safety policy, said improvements have been made in the Leapfrog methodology since the June report card, but there are still problems. It is possible that hospitals may look better or worse depending on the variability in their answers on the Leapfrog survey, as opposed to their actual safety records.

Meanwhile, hospitals that did not use the Leapfrog survey seemed to get penalized. Those answering a Leapfrog survey question about implementation of CPOE were eligible for up to 100 points, but hospitals answering a similar question on the AHA survey could earn only 65 points, Foster said.

“We would prefer that it be an even playing field,” she said. That highly regarded hospitals such as Cleveland Clinic got poor grades should cause concern about Leapfrog's methodology, Foster added.

“It does make one question whether the data that's being used is in fact as accurate as it should be,” she said. “It's that sort of face-validity question.”

Previous research has found that hospital report cards are inconsistent in how they evaluate care, are hard for patients to understand and use, and fail to demonstrably improve quality or safety.

Grading will continue

Yet Leapfrog is not backing down, said Leah Binder, its president and CEO.

“The expert panel will reconvene in the new year, and they'll just continue on a regular basis to review the methodology for calculating the hospital safety score, and we will continue to be putting out grades for all the general hospitals,” she said. “Our board is 100% committed to this, and our expert panel is behind it, and we're full speed ahead.”

Hospitals unhappy with their grades should inspect their scores and implement quality improvements, Binder said.

“We've used publicly available, well-vetted, focused safety measures, and we've made it 100% transparent so that hospitals can go in themselves and see how did they get that C,” she said. “We're not selling magazines, and this is not a one-time deal. This is about an ongoing way of improving the performance and safety of our hospitals. I think that's caught their attention.”

Fifty-eight percent of hospitals got the same grade in November as they got in June, while 34% saw their grades change up or down by one level. An additional 8% saw them change by two or more grade levels. The Leapfrog grades will be handed out annually from now on, the organization said.

In December, Leapfrog announced a consulting arrangement with Johns Hopkins' Armstrong Institute for Patient Safety and Quality in Baltimore. The Leapfrog grades are not the final word on hospital safety, said Matt Austin, PhD, an instructor at the institute, who formerly directed Leapfrog's hospital survey.

“This should have a physician at least double-check him- or herself to say, 'Oh, am I admitting patients to a hospital that maybe doesn't have the best patient safety record? Should I be working with hospital administrators to see what they're doing to improve safety?' ” said Austin, an expert on quality measurement. “To me, these grades are a conversation starter about patient safety, a way for doctors to talk to hospitals where they have patients, and a way for patients to talk with doctors.”