Overcoming Biases in Assessing CBO Effectiveness

Introduction

As a community-based organization (CBO), you may think your intervention is failing when, in fact, it is wildly successful. The illusion of failure can occur when a comparison is made between post-enrollment medical spending on complex patients (clients) and pre-enrollment spending levels. Health sector partners are motivated by the prospect of CBO services reducing spending on their patients’ inpatient and emergency department care. Evidence that medical expenditure is higher for a cohort of clients after enrollment is not encouraging. However, it is not necessarily indicative of failure. The counterintuitive result may stem from a bias that can creep into the evaluation. The tendency is called progression from the mean. It is common for clients to be selected for intervention based on new risk rather than old cost.

When dealing with a complex population that requires social services, the term ‘high-risk, high-cost’ is often used to define the target population. However, high-risk clients do not necessarily have high costs, and vice versa.

High-Risk Eligibility Criteria Can Lead to Understating CBO Effectiveness

A criterion for inclusion into a program that addresses the social impacts of health is for the complex patient to be described as high-risk. They may not exhibit high medical utilization, but will do so soon. Such patients might be referred by physicians who judge them to be on the verge of fully expressing severe and underlying risk factors. High-risk as the eligibility criterion is prospective. Enrollment is based on what spending is expected to be in the future. Suppose an evaluation methodology is based on comparing medical spending before enrollment to form a baseline against which program success can be measured. In that case, a bias will affect conclusions about the program’s impact. The measured cost avoidance probably understates the actual financial benefits. The reason for the bias, called progression from the mean, is that high-risk patients may be newly classified because they have begun to exhibit symptoms and acquire morbidities that are precursors to high future spending.

Consequently, spending in prior periods understates the likely spending in future ones. You would expect to see medical spending rising over time. However, that would not necessarily represent a failure of the intervention. The program might have succeeded in reducing the spending that would have otherwise occurred if usual care were provided. It may have succeeded in delaying the upward trajectory of spending and saving money.

High-Cost as the Eligibility Criterion Can Lead to Overstating CBO Effectiveness

The second common basis for inclusion into a program is when the person is deemed high-cost. High-cost is usually a retrospective criterion that depends on exhibiting a high total cost of care in a preceding period. The financial department of a health system is likely to select on this basis, as would a medical group. The bias that can result here is called regression or reversion to the mean. Regression to the mean illustrates that high spending tends to naturally revert to a level more closely approximating the mean or average. There was something unusual and unsustainable in the patient’s immediate past.

Choosing a retrospective criterion and using it to form a baseline against which program success can be measured can result in an overstatement of the financial benefits of a program to address the social impacts of health. The potential bias here stems from the realistic probability that a cost outlier might, in the subsequent period, revert closer to the norm, even in the absence of an intervention designed to reduce expense. Regardless of the program, some enrollees would not have exhibited sustained, high medical spending.

Both High-Cost and High-Risk as the Eligibility Criterion: Biases Cancel Out

Conceivably, if the population receiving the elevated level of care is an even mixture of high-cost and high-risk, the biases cancel themselves. The change in utilization from the baseline level (preprogram) would be a fair measure of the ROI. Of course, that cannot be counted on. A different approach is needed to measure ROI.

High-Risk as the Eligibility Criterion (But with a Different Baseline Measure)

An accurate ROI of a program requires comparing the actual post-program medical spending with what would otherwise have been spent under usual care. A control or comparison group for this evaluation is advantageous but impractical in most cases.

Another option is to predict future utilization under usual care, not based on prior utilization, but rather by assessing each person’s health risks and forecasting the cost of care for a person at that age, with those diseases, functional limitations, cognitive impairments, and social service needs. The predicted spending level then becomes the baseline. After the program has been implemented, reductions from that baseline will become accurate measurements for the ROI. Accurately forecasting future spending is relatively straightforward if the data is sufficient and the data analytics team can conduct statistical analysis.