Projections vs. Predictions, redux
I’ve written about the difference between projection models and forecasts/predictions before. Unfortunately, this is a distinction that people appear to remain thoroughly confused about.¹ In brief, recall that a forecast/prediction aims to make claims about what the future will look like, while projections are conditional hypotheticals about what would happen under certain scenarios.
Why am I raising this again? Because there are quite a few “modeling critics” out there who are increasingly vocal and resolutely wrong in the critiques they’re raising about model performance.
To illustrate, let me take the discussion briefly out of the realm of COVID-19. Let’s talk about hurricanes for a minute. Suppose a category 5 hurricane were bearing down on Miami, and FEMA came out with a projection that said, “If no one evacuates, as many as 10,000 people could die.” In response to that warning, let’s assume that 75% of the people in designated evacuation areas followed the warning and got out of harm’s way, and those who stayed behind were in places that were evaluated to be low risk. Let’s further assume that the storm stayed its course, and behaved more or less as the weather forecasters predicted. And all told, 25 people died, but this was accompanied by lots of property damage. Would people claim that FEMA’s projection was “wrong” because 10,000 people didn’t die? I would hope not. Instead critics would recognize that people responded to the warning in a way that changed the conditions. “If no one evacuates” never came to pass because people evacuated, so the conditions of that projection did not apply.
This is how projections are used. They identify expectations under specifiable scenarios, in hopes to motivate changes that will lead those scenarios to never come about. They should not be evaluated as though they were forecasts — especially not if their projections motivated behavioral changes that successfully alter the conditions in the desired ways.
In the case of COVID, these conditions could estimate effects “if people maintain normal activities and do ___,” where ___ could represent social distancing recommendations, mask wearing requirements, etc.
But, the hope is that seeing what those scenarios could bring about will help motivate people to change their behavior. E.g., if a model says “if no one wears a mask XX people would die of COVID,” we hope that people will wear masks to protect those XX people. And if people do change their behavior, it doesn’t make sense for modeling critics to then say “see, the models were wrong” when the conditions changed.
The encouraging thing is that in the past 6 months, people’s behaviors have changed. Here in Colorado for example, mobility data showed (see e.g., Fig 3) that people started acting as though we were under a stay at home order up to a week before that stay at home order was in place (i.e., people took up social distancing at high rates, some on their own). And people have been more compliant with mask wearing here than in many other states. In fact, the magnitudes of many of these condition changes have been greater than most assumed was possible at the outset of COVID-19.²
So, any suggestion that models based on scenarios without behavior change were “wrong” fundamentally misunderstands the purpose of such models in the first place. We never assumed people would do nothing. As my initial piece said, any model must be evaluated on the aims for which it was constructed. There are forecasting models out there. But scenario-based projections shouldn’t be treated as forecasts. Projections are intended to “play out” different hypothetical scenarios, often to guide what policy decisions are likely to have what types of effects.
If you want to see how well a projection performed, you could do one of two things. First you could ask what behaviors it was designed to motivate/inform changes about and see how well those conditions appear to have responded to the recommendations the model was designed to motivate. Second, you could find scenarios that align with the conditions that actually came about, and see how well the outcomes also aligned (but note that it’s often difficult to find exact combinations of scenarios appearing in the “real world”).
But if you want to compare the outcomes to reality, that should be reserved for assessing forecasts/predictions.
¹ I’ve decided to not call out particular offenders here.
² One of my colleagues told me of having a paper modeling some general levers of social distancing rejected by a journal prior to 2020, because one of their considered scenarios included school closures, and the editor simply couldn’t believe that would ever be an intervention that would be implemented.
