The Folly of Health Care Quality Studies

The folly of "quality studies" is the quality. One would think that there would be quality in studies published in the most prestigious journals or reported by the Institutes of Medicine. But you should be skeptical of any claim regarding quality. This is a basic requirement to understand what quality is and what it is not in application to health care.

The media constantly turns to the latest quality health care studies to attract attention and readership. Some are good reviews. Others do bashing of doctors, or certain types of providers. Some articles promote their favorites such as nurse practitioners or various innovations. What stimulated my attention this time is July is a Dangerous Hospital Month on Vox. After the BMJ article about medical errors, it is likely that we will continue to see more in this area. Doctor bashing has become quite popular, especially when journalists experience health care in certain ways.

What journalists and even quality experts tend to forget is that quality studies are actually quite limited because people are so complex and their interactions are even more complex.

People Factors Shape Outcomes Most of All

Quality Failure Is About Complexity in People

One person is complex. Two people interactions are many times more complex. Groups have astronomical numbers of interactions. Patient outcomes are shaped most of all by patient factors and people are quite different and difficult to capture in life as in studies.The number of differences and the degree of differences is beyond measure (lower income, lower education, health literacy, occupation, origin, parents, access barriers...). Providers are groups of people and are also very complex in interactions and difficult to define, much less assess.

When we do not know something, we guess, estimate, or assume. But what we do not often do is consider what shapes quality overall.

Patient behavior, social determinants, situations, and environments

...are the most powerful determinants of outcomes involving health or health studies. The complexity of such interactions is beyond measure, but this does not deter those bent on demonstrating their version of what shapes quality.

With People Factors Fixed in Place, Outcomes Are Fixed in Place

People factors are difficult to address, especially in a nation that divides into few well off and most worse off. The only rhyme or reason is that they usually line up better for those better off and worse for those worse off. This is not understood by many if not most researchers, who often fail to look for the reasons behind their research findings. The answer is quite simple. Researchers learn from mistakes such as poor assumptions. When they explore why they were wrong, they learn. If they find what they are looking for, they are not likely to question their findings. Science moves forward by failure more than by success.

People Factors Are Not Understood Because Most People Are Not Studied

The well off are often captured in studies. Studies are mainly about those in urban areas or seen by large institutions or the largest hospitals. We most commonly have studies that sometimes show differences, but these may not be relevant for most Americans.

Genetics

Genetic factors are present and are nearly impossible to comprehend or address at the current time. Genetics, situations, and environments also interact to make studies even more difficult.

Unknown

There are also unknown influences, which is why social sciences studies explain only 30 - 40% of the reasons and often less. But some difference has to be present to be of interest to the media. This is why no headline is accurate. To be accurate would mean a headline "We don't know about a most dangerous month being July, or teaching hospitals or rural hospitals or primary care medical homes but the few factors we can study indicate a small difference not likely relevant." In the interest of time and generating interest, the media coverages are not likely to be accurate, especially when limited to a few words.

Researcher Errors Magnified By the Media Coverage

Claims of Significance and Relevance Are Often Exaggerated

All the above factors that shape quality also interact with each other making it very difficult to study anything or separate relevance from background noise. Lack of awareness remains a problem for authors, journal editors, and those who shape policy or information. Assuptions and opinions in journals should not be printed or should be characterized as such, but that has not prevented printing.

Even though no one can claim proof, this does not prevent claims of proving this or that.
Also no journal or medial piece should indicate that all areas were controlled for.
There are no appropriate controls for the many dimensions of patients and their environments and their genetics and other unknown reasons.

Researchers Hold All the Cards

Anyone who has played with regressions understands how the findings change with the variables and the controls.The authors can structure the question, use data not collected specific to the question, ignore data that might be relevant, select the controls that change the outcomes, or fail to indicate other variables which might eclipse the one that they have decided is most important to indicate.

The Media Chooses the Studies to Magnify or Exaggerate

Studies have actually demonstrated the distortions in the media - subject to popularity, advertising, and other dollar focus.

The Damage Done: Quality Studies Are Often About People Differences

Usually "quality" studies demonstrate differences, but the real differences are in the two populations being compared. In the US, there is an assumption that rural or lower volume contributes to lesser outcomes. If you look for this, you will find it - sometimes. But you will never really know if this was due to "Rural" or "lower volume" or "smaller" because such studies are often making a major research error of comparing apples to oranges.

Rural hospitals or hospitals with lower volume are often included in studies even though they are different in many ways - patients, resources, workforce, cost of delivery, types of services, and funding.
Resident work hours were instituted to "improve" the quality of patient care in teaching hospitals. Of course the patient populations remained the same as were the outcomes. The situations were the same. Common sense might indicate that residents were limited in impact due to little time and others involved, but this has not stopped the studies including some that may be unethical.
Academic hospitals are also quite different from others and often from each other.
Nurse practitioner vs MD studies would be expected to have the same results - as long as the populations being compared are much the same.
Primary Care Medical Home studies are quite biased due to certain types of practices that choose PCMH and others do not - size, types of clinicians, financing, location, etc. The smaller independent practices with different types of clinicians and in different locations are compared with those more organized and different in other ways. The PCMH studies have been revealing about foundations that tend to fund researchers doing research that they support. Studies have also exposed the misinformation regarding the homes.

Doctor Bashing is Popular and Popular Gets Published

Studies often make it to publication, but there are major flaws. This is a nice review by Kip Sullivan in The Health Care Blog of what it takes to demonstrate that the findings are due to a particular physician.

[H]ere are several important determinants of validity of physician performance measures:

Is the measure fully controllable by the physician?

Is the measure properly adjusted for variation in the case-mix of patients among physicians?

Is the measure partially controlled by some other level of the system?….

Reliability assumes validity as a necessary precondition. Although the reliability calculations can still be performed for measures that are not valid, subsequent interpretation is problematic. [p. 17]

I think we can go beyond “problematic” in criticizing CMS’s proposal to use patient pools as small as 20. I believe “reckless” is the appropriate word.

What is rare

is to find research that exposes the high costs of PCMH or MACRA or EHR or those going against the bandwagon of innovation.
is to find articles exposing the limitations of innovative approaches
is a study that points out the limited outcomes.
is a study that points out how the increased costs diverted to non-delivery areas impacts those delivering care or the populations in need of care.
is any expert indicating the folly of poor pay for primary care, office, mental health (finally someone did so)

Despite the Inability to Discern Quality, Payment Designs Have Force EHR Based "Quality"

The limitations of EHRs are legendary. Even worse, the best studies have demonstrated lack of ability to discern quality. Quality study limitations mean that the best quality studies cannot discern quality physicians from the average physician (NY CABG studies) even with much more data and expense. The various quality report cards are even more limited in data and expense and mostly in inability to discern quality. CMS makes claims far beyond the literature and the findings of its consultants, making matters worse.

It is also why over a dozen studies have demonstrate that Pay for Performance results in less pay for providers that care for poor, less advantaged, rural, underserved, Community Health Center, or similar populations - but this has not stopped the discrimination from being implemented nationally.

Despite journal articles and RAND studies that raise caution about pay for performance, health policy has moved all the way to implementation via MACRA and CMS. This attempt to discern quality cannot discern quality and will be done at great cost and with impairment of health care productivity as reviewed in recent blogs.

Fun with Regressions Can Be Misunderstood

Sometimes fun with regressions is used by journals to generate much discussion. The BMJ did its job with the recent medical error study according to experts.

But a side effect of medical error is that physicians are blamed in medical error studies from To Err is Human to the present.

And most importantly, countless billions have been invested
in the attempt to improve quality without the ability to measure or discern quality.

The Real Story in Health Care Quality Is Often What Is Rarely Studied

Ask yourself about influences that might impact quality

Government decisions to cut payment have numerous impacts on the ability to deliver care, but are rarely studied. These studies make claims of cost savings, but only in one area while it can be more costly in other areas.
Administrations facing cuts or attempting to focus on profits have cut down on the budget areas with the most dollars - personnel. This has resulted in nurse staffing ratios that might just be too low to maintain quality and patient safety.
Where are the studies of outcomes from dumping patients out too early or without regard to readiness at home, or in areas with low resources or missing primary care.

Studies Can Mean Little for Most Americans

We also do not study representative populations and we often do not even know if various treatments (prolonged chemotherapy regimens) actually work on certain populations. Many Americans have jobs without sick leave pay and few have sick leave or benefits that can cover prolonged illness. This makes it difficult to tolerate the full treatment due to job, access, transportation, health literacy, or other issues.

This is one of many reasons why a cancer moonshot is likely to benefit the few that can afford, tolerate, and understand complex treatments. Meanwhile most Americans, especially government associated (Medicare, Medicaid, Metallic, fixed income, no pay) need basics like better nutrition, community resources, mental health, primary care, community outreach...

We do know that certain populations respond better with more support as seen in support for care at the end of life and home visit demonstrations. This would have been discovered long ago if the focus was placed on care rather than costs. Costs are cut from those least able to defend themselves and spending goes to those most organized. This also helps shape studies and outcomes in favor of the usual suspects. Over decades of time, this also helps layer Americans into Advantaged and Disadvantaged as health spending shapes one group to better off and most Americans worse in income, jobs, situations, and other determinants.

In fact the one defining characteristic of American health care is less for most Americans and more designed to benefit few. There has been a strong bias against those who cannot speak for themselves who are less organized.

Read Between the Lines in Journals and in Media Articles

So next time your review a quality study from the smallest to major journals to IOM studies, you can be sure of one thing. Much of what is discussed is not really understood well.

There are also some good studies regarding how authors can manipulate the findings by their choices - and how social research is often difficult to replicate.

There are many reasons to manipulate or to avoid critical thinking. There are few reasons to have high integrity.

Search This Blog

Basic Health Access