Overcoming Biases in Assessing CBO Effectiveness

Introduction

As a Community Based Organization (CBO), you may think your intervention is failing when in fact, it is wildly successful. The illusion of failure can occur when a comparison is made between post-enrollment medical spending on complex patients (clients) and pre-enrollment spending levels. Health sector partners are motivated by the prospect of CBO services reducing spending on their patients’ inpatient and emergency department care. Evidence that medical expenditure is higher for a cohort of clients after enrollment is not encouraging. However, it is not necessarily indicative of failure. The counterintuitive result may stem from a bias that can creep into the evaluation. The tendency is called progression from the mean. It is a common phenomenon when clients are selected to receive the intervention based on new risk rather than old cost.

When dealing with a complex population that needs social services, the term high risk-high cost is often misled to define the target population. But high-risk clients are not necessarily high cost and vice versa.

High-Risk Eligibility Criteria Can Lead to Understating CBO Effectiveness

A common criterion for inclusion into a program that addresses the social impacts of health is for the complex patient to be described as high-risk. They may not currently exhibit a high level of medical utilization but will do so soon. Such patients might be referred by physicians who judge them to be on the verge of fully expressing severe and underlying risk factors. High-risk as the eligibility criterion is prospective. Enrollment is based on what spending is expected to be in the future. Suppose an evaluation methodology is based on comparing the actual medical spending from a period before enrollment to form a baseline against which program success can be measured. In that case, there is a bias that will affect conclusions about the program’s impact. The measured cost avoidance probably understates the actual financial benefits. The reason for the bias, called progression from the mean, is that high-risk patients may be newly classified because they have just begun to exhibit symptoms and acquire morbidities that are precursors to high future spending.

Consequently, spending in prior periods understates the likely spending in future ones. You would expect to show medical spending rising over time. But that would not necessarily represent a failure of the intervention. The program might have succeeded in reducing the spending that would have otherwise occurred if usual care were provided. It may have succeeded in delaying the upward trajectory of spending and saving money in the process.

High-Cost as the Eligibility Criterion Can Lead to Overstating CBO Effectiveness

The second common basis for inclusion into a program is when the person is deemed high cost. High cost is usually a retrospective criterion that depends on exhibiting a high total cost of care in a preceding period. The financial department of a health system is likely to select on this basis than would a medical group. The bias that can result here is called regression or reversion to the mean. Regression to the mean illustrates that high spending tends to naturally revert to a level more closely approximating the mean or average. That there was something unusual and not sustainable in the patient’s immediate past.

Choosing a retrospective criterion and using it to form a baseline against which program success can be measured can result in an overstatement of the financial benefits of a program to address the social impacts of health. The potential bias here stems from the realistic probability that a cost outlier (meaning someone at the right tail of the cost distribution) might, in the subsequent period, revert closer to the norm, even in the absence of an intervention designed to reduce expense. A certain percentage of enrollment enrollees would not have exhibited sustained, high medical spending, regardless of the program.

Both High-Cost and High-Risk as the Eligibility Criterion: Biases Cancel Out

Conceivably, if the population receiving the elevated level of care is an even mixture of both high-cost and high-risk, the biases cancel themselves. The change in utilization from the baseline level (preprogram) would be a fair measure of the ROI. That, of course, cannot be counted on. What is needed is a different approach to measure the ROI.

High-Risk as the Eligibility Criterion (But with a Different Baseline Measure)

What is required for an accurate ROI of a program is for the actual post-program medical spending compared with what otherwise would have been spent under usual care. It is advantageous to have a control or comparison group for this evaluation, but it is impractical in most cases.

Another option is to predict the future utilization under usual care, not based on prior utilization, Instead, by assessing each person’s health risks and forecasting what it would cost to care for a person at that age, with those diseases, with the functional limitations, cognitive impairments, and social service needs. The predicted spending level then becomes the baseline. After the program has been in place, reductions from that baseline become the accurate measurements for the ROI. The task of accurately forecasting future spending is relatively straightforward if the data is sufficient and the data analytics team can conduct statistical analysis.