Physician profiling found wrong 22% of the time

Physicians are at a strikingly high risk of being miscategorized when rated by cost-profiling programs, according to research mimicking the types of tiering programs health plans have used.

The study, funded in part by the U.S. Dept. of Labor and conducted by researchers at RAND Corp., outlined the outcome of a mock profiling program using claims from four Massachusetts health plans. The authors created a sample program similar to ones health plans have used to rate physicians' efficiency.

Using the insurers' claims from 2004 and 2005, researchers found that the model two-tier program misclassified as many as 22% of physicians, and that about two-thirds of physicians' designations would be less than "optimally reliable."

"Consumers, physicians and purchasers are all at risk of being misled by the results produced by these tools," concludes the study, which appeared in the March 18 New England Journal of Medicine.

John L. Adams, PhD, the RAND statistician who led the research team, said his aim was to add some hard evidence to the debate between health plans and physicians, which has been contentious and often ended in stalemate.

The study looked at cost profiling, not quality measurement, though many health plans merge the two into a single "efficiency" or "performance" score. Physicians have complained that those scores have been based more heavily on cost than quality.

The study's authors measured reliability by using a statistical analysis that accounted for a normal amount of variation or "noise" associated with each physician, then compared that with the variability that showed up in the tiering system. The model program produced a level of variability that meant many physicians ended up misclassified.

"What does it mean for a tool to be good enough?" Adams said. "No tool is going to be perfect ... I think there are people that look at a 25% misclassification rate and say, 'Great, better than what we had yesterday,' but almost no physician would find a 25% misclassification rate acceptable."

He said he didn't want to take a position on what percentage misclassification might be acceptable. But others did not hesitate to do so.

"It's critically important that patients and physicians get clear, accurate information about the cost and quality of health care. But this report, produced by an independent, renowned research firm, clearly demonstrates that these profiling programs fail to accomplish those goals," Mario Motta, MD, a cardiologist in Salem, Mass., and president of the Massachusetts Medical Society, said in a statement.

The Massachusetts Medical Society has sued the state's Group Insurance Commission, which purchases health coverage on behalf of state and some municipal employees, over what physicians believe are serious flaws in the GIC's physician profiling program.

The society owns the NEJM, and the authors disclosed past grant support from the organization, as well as grant support and consulting fees from the American Medical Association and other medical societies.

Dolores Mitchell, GIC executive director, who in the past has defended the profiling initiative as an imperfect but necessary program, did not respond to requests for comment.

The AMA pointed to the study's results as support for its long-held position that cost-profiling programs aren't designed well enough to give either patients or physicians good information.

"Inaccurate information can erode patient confidence and trust in caring physicians, and disrupt patients' longstanding relationships with physicians who have cared for them for years," AMA President J. James Rohack, MD, said in a statement.

He called on health insurance companies to "abandon flawed physician evaluation and ranking programs, and join with the AMA to create constructive programs that produce meaningful data for increasing the quality and efficiency of health care."

The AMA-led Physician Consortium for Performance Improvement is a project that focuses on outlining useful and reliable clinical measures of quality. The PCPI has introduced 266 model quality measures but has not published model cost-profiling methodology.

AMA policy calls for rating systems to be transparent and based on evidence-based quality measures rather than cost. The AMA also urges health plans to give physicians time to review the data behind such ratings and allow physicians to appeal if they disagree with the basis for their designation or score.

Robert Zirkelbach, spokesman for health insurance trade group America's Health Insurance Plans, said the field of physician ratings is "an evolving area."

"The report does highlight what health plans agree are two important areas that need to be addressed," he said. "First, there has to be an adequate sample size, and two, that you're attributing performance to the right physicians. Doing this in collaboration with physicians is key, so patients can have the peace of mind knowing the data is reliable and something they can trust."

Some health plans have been forced to scale back cost-profiling programs or reinvent them, most notably in New York. In that state, Attorney General Andrew Cuomo in 2007 sued insurers doing business there, demanding they abandon cost-based profiling programs he said were disguised as quality measurements.

UnitedHealth Group, Cigna, WellPoint, Aetna and several smaller insurers, while not admitting any wrongdoing, settled with the attorney general's office, agreeing to disclose the basis of physician tiering programs and tie them to recognized standards of quality, not solely cost. The largest plans also pledged to take those changes national rather than limiting them to the New York market.

In 2008, most of the country's largest insurers signed on to a charter agreement developed by the Consumer-Purchaser Disclosure Project, a group of businesses and consumers who demanded more transparent and meaningful quality measurement. The insurers agreed to have their tiering programs audited by the nationally recognized organization and to work toward using the same measures industrywide to gauge physician performance.