The quality quandary: The problems with ratings

Can a number tell the world how good a doctor is? A lot of people are counting on it. The government, health plans and employers are demanding a way to quantify the value they're getting for their health care dollars. Patients, faced with higher co-pays and deductibles and increasingly invested in health savings accounts, want reliable public ratings of doctors' performance.

Only a third of physicians have access to any data about their own clinical performance, and just a quarter have the electronic medical records systems considered essential for continuous quality improvement, according to a 2003 Commonwealth Fund survey of doctors. Still, a small but growing number of physicians who have assessed the quality of care they deliver as part of maintenance-of-certification and pay-for-performance programs see quality measurement as helpful to care better for their patients.

But many physicians, embittered by early and clumsy attempts by outsiders to gauge their performance, have deep concerns about the accuracy and fairness of quality measurement. With more than 100 pay-for-performance programs up and running, according to the Leapfrog Group, and the Centers for Medicare & Medicaid Services asking physicians to voluntarily report quality data, it seems quality measurement is here to stay.

Yet as physician groups take over the reins in developing quality measures for payers and the public to use, a quandary presents itself: Could attempts to improve quality by measuring it have the perverse effect of punishing doctors who care for the most vulnerable patient populations?

Skeptical doctors say the medical experts, academicians, methodologists and statisticians charged with developing quality measures must be mindful of their potential impact on practicing physicians and the patients they serve.

Concerns about patient outcomes

Randall Maxey, MD, PhD, approves of the quality measurement idea in general but worries about whether its use could wind up penalizing physicians who practice in underserved areas. While most measures gauge how often physicians deliver clinically recommended care, a few very important ones hold doctors accountable for their patients' outcomes.

"It's going to be a lot easier to treat a little old lady from Beverly Hills," said Dr. Maxey, an Inglewood, Calif., nephrologist who co-chairs the Commission to End Health Care Disparities and a National Medical Assn. former president. "Some communities are more compliant and more health-literate and have more resources to influence outcomes than others. I may treat you exactly correctly and give you the right pills, but if you have to choose between buying pills and giving your baby milk, that drug may lose out and my performance may be judged as poor because of it."

Roy M. Poses, MD, shares Dr. Maxey's concern.

"Outcomes are determined not just by what the physician does but by how sick the patient is, what his or her other characteristics are, and to some extent, by chance," said Dr. Poses, a clinical associate professor of medicine at Brown Medical School who practices at Internal Medicine Health Associates in Taunton, Mass. "If you don't control for patient characteristics, you can have a perverse system."

While some efforts have been made to adjust outcome-based measurement for illness severity or patient population, Dr. Poses said, "very few outcome measurement systems control for patient characteristics so well that I would not worry about them having unintended consequences."

Leading figures in the quality measurement movement concede that physicians have valid concerns about how much they can control patients' self-management of cholesterol levels, blood pressure or hemoglobin A1c.

"It's recognized by everyone that these are not measures that are under the control of an individual clinician," said Janet Corrigan, PhD, CEO of the National Quality Forum. "But there are important things a primary-care provider can to do to encourage patients to adopt the right behaviors." Physicians who follow a model of chronic illness care that incorporates reminder systems, patient education and other elements can achieve better results, she said.

Ultimately, "the patient controls the compliance," said Nancy Nielsen, MD, PhD, Speaker of the AMA House of Delegates. "But we can do a better job of helping patients take care of themselves," said Dr. Nielsen, who represents the AMA on the Ambulatory Care Quality Alliance, or AQA, a coalition of physicians, researchers and health insurers pilot-testing a single set of measures and data standards. "That's why the AMA got into the performance measure business years before anybody in Congress thought of pay for performance."

Greg Pawlson, MD, MPH, said that physicians' concerns about outcome-based measures are legitimate but that by definition the problem is not as widespread as many believe. "Not everybody's patients can be sicker," said Dr. Pawlson, executive vice president at the National Committee for Quality Assurance.

There are extreme cases where physicians could be penalized for practicing in underserved areas, Dr. Pawlson said. In those instances, he said, it would be fairer to compare physicians who mostly care for Medicaid or uninsured patients with each other, rather than to practices at large.

Physicians may take an unrealistic view of how sick their patients actually are, said Bruce Bagley, MD, medical director of quality improvement at the American Academy of Family Physicians. "If I've got 100 diabetics in my office, their outcomes are going to be on a bell-shaped curve. ... But physicians will tend to focus on the worst-case scenario and generalize to their whole diabetic population."

Taking control

Outcome-based measures aren't the only area where patient noncompliance is a factor, according to Chuck Kilo, MD, MPH, a Portland, Ore., internist at GreenField Health.

"Do I get dinged if a diabetic chooses not to have their A1c tested, or is my recommendation sufficient to get credit?" asked Dr. Kilo, a senior fellow at the Institute for Healthcare Improvement and a quality-measurement advocate. "If so, then that leaves a lot of room for gaming the system."

Most measures allow for certain exclusions including medical contraindications or patient refusals, but another problem is that quality measurement based on administrative claims data often hasn't captured these nuances -- either a test was ordered or it wasn't.

"Health plans have been measuring practices and sending data back for a long time, and most doctors would throw them in the circular file," Dr. Kilo said. "Somewhere between 10% or 50% of the patients they have listed as mine are not mine. It doesn't take a whole lot of erroneous data built into it for doctors to write off the whole thing."

Dr. Kilo said it's time for physicians to take control of measuring their own performance.

According to Christine K. Cassel, MD, the debate over the potential impact of quality measurement is "kind of going on in the stratosphere because most physicians don't have the infrastructure in their office to measure quality."

Most physicians lack the systems capacity to even find out the denominator at play in many quality measures, such as how many hypertensive patients they have overall, said Dr. Cassel, board president of the American Board of Internal Medicine, which this year began requiring physicians to report on their performance to maintain certification.

Dr. Bagley cites ABIM's move as part of a shift toward evaluating physicians on their performance as well as their pedigree. "It's not only what I know," he said. "It's what I do."

Still, some suspect the push for quality measurement and the specter of pay-for-performance is more about cutting costs than helping patients.

"I try to help patients get the best outcomes as best I can," said Dr. Poses, president of the Foundation for Integrity and Responsibility in Medicine, a small nonprofit that supports representative, transparent, ethical and responsible health care governance. "If the measurement systems or incentives are really designed to save costs for health plans, they may push physicians not to do that which would be the best for the patient in terms of clinical care and clinical outcomes. The devil is in the details."

For its part, the AMA says it is focused on developing quality measures that are fair and useful for physicians, patients and payers. The AMA-convened Physician Consortium for Performance Improvement has doubled its staff and is working with the NCQA and data-collection firm Mathematica to develop 140 quality measures covering 34 clinical areas for CMS' Physician Voluntary Reporting Program by the end of this year. At press time, 93 had been completed.

The battle for physician control over quality measurement may be nearing an end, but the fight over how the quality data will be used by payers and the public is just beginning.

"We can say all we want," Dr. Nielsen said, referring to the pay-for-performance principles passed by the House of Delegates in June 2005. "It's how these measures get implemented by others that's going to determine whether it's about quality or not."