The quality quandary: The problems with ratings

As the movement to measure quality forges ahead, some physicians worry that their patients could be left behind.

By Kevin B. O’Reilly — Posted May 22, 2006

Print  |   Email  |   Respond  |   Reprints  |   Like Facebook  |   Share Twitter  |   Tweet Linkedin

Can a number tell the world how good a doctor is? A lot of people are counting on it. The government, health plans and employers are demanding a way to quantify the value they're getting for their health care dollars. Patients, faced with higher co-pays and deductibles and increasingly invested in health savings accounts, want reliable public ratings of doctors' performance.

Only a third of physicians have access to any data about their own clinical performance, and just a quarter have the electronic medical records systems considered essential for continuous quality improvement, according to a 2003 Commonwealth Fund survey of doctors. Still, a small but growing number of physicians who have assessed the quality of care they deliver as part of maintenance-of-certification and pay-for-performance programs see quality measurement as helpful to care better for their patients.

But many physicians, embittered by early and clumsy attempts by outsiders to gauge their performance, have deep concerns about the accuracy and fairness of quality measurement. With more than 100 pay-for-performance programs up and running, according to the Leapfrog Group, and the Centers for Medicare & Medicaid Services asking physicians to voluntarily report quality data, it seems quality measurement is here to stay.

Yet as physician groups take over the reins in developing quality measures for payers and the public to use, a quandary presents itself: Could attempts to improve quality by measuring it have the perverse effect of punishing doctors who care for the most vulnerable patient populations?

Skeptical doctors say the medical experts, academicians, methodologists and statisticians charged with developing quality measures must be mindful of their potential impact on practicing physicians and the patients they serve.

Concerns about patient outcomes

Randall Maxey, MD, PhD, approves of the quality measurement idea in general but worries about whether its use could wind up penalizing physicians who practice in underserved areas. While most measures gauge how often physicians deliver clinically recommended care, a few very important ones hold doctors accountable for their patients' outcomes.

"It's going to be a lot easier to treat a little old lady from Beverly Hills," said Dr. Maxey, an Inglewood, Calif., nephrologist who co-chairs the Commission to End Health Care Disparities and a National Medical Assn. former president. "Some communities are more compliant and more health-literate and have more resources to influence outcomes than others. I may treat you exactly correctly and give you the right pills, but if you have to choose between buying pills and giving your baby milk, that drug may lose out and my performance may be judged as poor because of it."

Roy M. Poses, MD, shares Dr. Maxey's concern.

"Outcomes are determined not just by what the physician does but by how sick the patient is, what his or her other characteristics are, and to some extent, by chance," said Dr. Poses, a clinical associate professor of medicine at Brown Medical School who practices at Internal Medicine Health Associates in Taunton, Mass. "If you don't control for patient characteristics, you can have a perverse system."

While some efforts have been made to adjust outcome-based measurement for illness severity or patient population, Dr. Poses said, "very few outcome measurement systems control for patient characteristics so well that I would not worry about them having unintended consequences."

Leading figures in the quality measurement movement concede that physicians have valid concerns about how much they can control patients' self-management of cholesterol levels, blood pressure or hemoglobin A1c.

"It's recognized by everyone that these are not measures that are under the control of an individual clinician," said Janet Corrigan, PhD, CEO of the National Quality Forum. "But there are important things a primary-care provider can to do to encourage patients to adopt the right behaviors." Physicians who follow a model of chronic illness care that incorporates reminder systems, patient education and other elements can achieve better results, she said.

Ultimately, "the patient controls the compliance," said Nancy Nielsen, MD, PhD, Speaker of the AMA House of Delegates. "But we can do a better job of helping patients take care of themselves," said Dr. Nielsen, who represents the AMA on the Ambulatory Care Quality Alliance, or AQA, a coalition of physicians, researchers and health insurers pilot-testing a single set of measures and data standards. "That's why the AMA got into the performance measure business years before anybody in Congress thought of pay for performance."

Greg Pawlson, MD, MPH, said that physicians' concerns about outcome-based measures are legitimate but that by definition the problem is not as widespread as many believe. "Not everybody's patients can be sicker," said Dr. Pawlson, executive vice president at the National Committee for Quality Assurance.

There are extreme cases where physicians could be penalized for practicing in underserved areas, Dr. Pawlson said. In those instances, he said, it would be fairer to compare physicians who mostly care for Medicaid or uninsured patients with each other, rather than to practices at large.

Physicians may take an unrealistic view of how sick their patients actually are, said Bruce Bagley, MD, medical director of quality improvement at the American Academy of Family Physicians. "If I've got 100 diabetics in my office, their outcomes are going to be on a bell-shaped curve. ... But physicians will tend to focus on the worst-case scenario and generalize to their whole diabetic population."

Taking control

Outcome-based measures aren't the only area where patient noncompliance is a factor, according to Chuck Kilo, MD, MPH, a Portland, Ore., internist at GreenField Health.

"Do I get dinged if a diabetic chooses not to have their A1c tested, or is my recommendation sufficient to get credit?" asked Dr. Kilo, a senior fellow at the Institute for Healthcare Improvement and a quality-measurement advocate. "If so, then that leaves a lot of room for gaming the system."

Most measures allow for certain exclusions including medical contraindications or patient refusals, but another problem is that quality measurement based on administrative claims data often hasn't captured these nuances -- either a test was ordered or it wasn't.

"Health plans have been measuring practices and sending data back for a long time, and most doctors would throw them in the circular file," Dr. Kilo said. "Somewhere between 10% or 50% of the patients they have listed as mine are not mine. It doesn't take a whole lot of erroneous data built into it for doctors to write off the whole thing."

Dr. Kilo said it's time for physicians to take control of measuring their own performance.

According to Christine K. Cassel, MD, the debate over the potential impact of quality measurement is "kind of going on in the stratosphere because most physicians don't have the infrastructure in their office to measure quality."

Most physicians lack the systems capacity to even find out the denominator at play in many quality measures, such as how many hypertensive patients they have overall, said Dr. Cassel, board president of the American Board of Internal Medicine, which this year began requiring physicians to report on their performance to maintain certification.

Dr. Bagley cites ABIM's move as part of a shift toward evaluating physicians on their performance as well as their pedigree. "It's not only what I know," he said. "It's what I do."

Still, some suspect the push for quality measurement and the specter of pay-for-performance is more about cutting costs than helping patients.

"I try to help patients get the best outcomes as best I can," said Dr. Poses, president of the Foundation for Integrity and Responsibility in Medicine, a small nonprofit that supports representative, transparent, ethical and responsible health care governance. "If the measurement systems or incentives are really designed to save costs for health plans, they may push physicians not to do that which would be the best for the patient in terms of clinical care and clinical outcomes. The devil is in the details."

For its part, the AMA says it is focused on developing quality measures that are fair and useful for physicians, patients and payers. The AMA-convened Physician Consortium for Performance Improvement has doubled its staff and is working with the NCQA and data-collection firm Mathematica to develop 140 quality measures covering 34 clinical areas for CMS' Physician Voluntary Reporting Program by the end of this year. At press time, 93 had been completed.

The battle for physician control over quality measurement may be nearing an end, but the fight over how the quality data will be used by payers and the public is just beginning.

"We can say all we want," Dr. Nielsen said, referring to the pay-for-performance principles passed by the House of Delegates in June 2005. "It's how these measures get implemented by others that's going to determine whether it's about quality or not."

Back to top


Process vs. outcome

Quality measures attempt to express how well physicians are adhering to widely accepted clinical guidelines. The denominator in these equations of care is the total number of patients clinically eligible for a given test, treatment or type of care. The numerator can be one of two things. Usually, it measures how often physicians implement certain processes of care; some numerators hold doctors accountable for actual patient outcomes. The National Committee for Quality Assurance has 10 measures for diabetes care. Four are process-based; six measure patient outcomes. Here's an example of each type:

Quality measure How a physician is scored
Testing (Process) Number of patients who had one or more HbA1c tests during the measurement year divided by the total number of diabetic patients ages 18-75 that the physician sees.
Poor control (Outcome) Number of patients whose most recent HbA1c level during the measurement year is greater than 9% divided by the total number of diabetic patients ages 18-75 that the physician sees.

Source: National Quality Forum-Endorsed Voluntary Consensus Standards for Physician-Focused Ambulatory Care, October 2005

Back to top

Who's measuring quality?

Several national organizations are involved in setting standards for how physician quality is, or will be, evaluated. Among them:

The National Committee for Quality Assurance

Who they are: The committee (link) was founded in 1990 with support from large employers and the managed care industry.

What they evaluate: Physician quality at the health-plan level via its widely used Health Plan Employer Data and Information Set. Administrative claims of about 70 million patients, or 85% of all HMO enrollees, are used to measure physician performance. The Centers for Medicare & Medicaid Services requires HMOs to submit Medicare HEDIS data.

Physician Consortium for Performance Improvement

Who they are: An AMA-convened group that includes representatives from more than 70 national and specialty medical societies, NCQA, CMS, the Agency for Healthcare Research and Quality and the Joint Commission on Accreditation of Healthcare Organizations.

What they evaluate: At press time, the consortium (link) had developed 93 measures of physician quality covering 15 conditions, ranging from asthma to osteoarthritis. CMS awarded a contract to the consortium, NCQA and data collection firm Mathematica to develop about 140 measures covering 34 clinical areas by year's end. By December 2007, the consortium's measures should cover the majority of Medicare spending on physician services.

National Quality Forum

Who they are: Incorporated in 1999, the group doesn't develop its own measures, but brings together consumers, employers, health plans, researchers, physicians and others to endorse quality measures.

What they've done: NQF (link) so far has endorsed 36 quality measures for ambulatory physician care; 24 were developed by the Physician Consortium for Performance Improvement, while the other dozen came from the NCQA.

Back to top

Assessing quality

The era of measuring physician performance is likely here to stay. Drawn from a range of perspectives, experts say these are some of the pros and cons of quality measurement.


  • Allows physicians to spot areas that need improvement and lets them use the knowledge to make changes.
  • Lets payers reward physicians who perform better with more referrals and bonus pay.
  • Empowers consumers to choose physicians on the basis of both quality and cost, particularly with the number of patients using health savings accounts at 3.2 million and growing. Increased competition for patients will lead to overall quality improvement.


  • Quality measurement can be inaccurate if it is based on administrative claims data that do not capture the entirety of a patient's circumstances.
  • Compliance can be costly. Either a practice must do timely, expensive chart reviews or spend heavily on electronic medical record systems.
  • Quality measures can unfairly represent physicians' performance. For example, outcome-based measures, especially those that do not adjust for illness severity, can penalize physicians who see patients who are sicker or face economic, cultural and linguistic barriers to complying with doctors' orders.
  • When paired with pay-for-performance, quality measurement could merely reward larger practices that have made costly technology investments and practices in higher-income areas. Perversely, quality measurement could divert resources away from already underserved areas.

Back to top



Read story

Confronting bias against obese patients

Medical educators are starting to raise awareness about how weight-related stigma can impair patient-physician communication and the treatment of obesity. Read story

Read story


American Medical News is ceasing publication after 55 years of serving physicians by keeping them informed of their rapidly changing profession. Read story

Read story

Policing medical practice employees after work

Doctors can try to regulate staff actions outside the office, but they must watch what they try to stamp out and how they do it. Read story

Read story

Diabetes prevention: Set on a course for lifestyle change

The YMCA's evidence-based program is helping prediabetic patients eat right, get active and lose weight. Read story

Read story

Medicaid's muddled preventive care picture

The health system reform law promises no-cost coverage of a lengthy list of screenings and other prevention services, but some beneficiaries still might miss out. Read story

Read story

How to get tax breaks for your medical practice

Federal, state and local governments offer doctors incentives because practices are recognized as economic engines. But physicians must know how and where to find them. Read story

Read story

Advance pay ACOs: A down payment on Medicare's future

Accountable care organizations that pay doctors up-front bring practice improvements, but it's unclear yet if program actuaries will see a return on investment. Read story

Read story

Physician liability: Your team, your legal risk

When health care team members drop the ball, it's often doctors who end up in court. How can physicians improve such care and avoid risks? Read story

  • Stay informed
  • Twitter
  • Facebook
  • RSS
  • LinkedIn