Corona model for PA just significantly improved

[Grant Green]

In early April at one point, IMHE were predicting around 760 deaths in the state as a total. It’s only June 11 and they were off by more than an order of magnitude. Most people would reasonably conclude there was and probably is something wrong with the model.

I've been through this multiple times on this board, but will give it one more try. The models are not giving a "prediction". They are providing a range of possible scenarios (deaths in this case) and a mean value. The range is typically mean +/- 1 standard deviation (not sure if it is exactly 1 SD in this case). The "mean" should not be confused with a "prediction".
Furthermore, the models are constantly adapting based on real time data. Due to the great number of unknowns and variables involved with a new virus, the range of possibilities is understandably large.

 

[UncleLar]

It's quite irresponsible for states to lump antibody tests in with their overall case counts.

I wouldn't call it irresponsible. The issue is that there is no coordinated federal reporting guidelines (ostensibly, because the feds didn't want to burden the states with having to redesig their data collection systems in order to be standardized across the country). The states collect data that they feel that is useful for them then further what they have collected on to the CDC. Counting antibody tests is important for several reasons. For example, it's a way of detecting how far along you are toward getting herd immunity within a paraticular community. It's important for states to know that at a local level. It's maybe not as important at a national level.

 

[interrobang]

I'm not saying don't count and publish the antibody tests. But don't lump them in with the active case counts (that the media loves to run with to prove we're all still doomed) when they don't accurately portray active case counts.

 

[The Spin Meister]

I've been through this multiple times on this board, but will give it one more try. The models are not giving a "prediction". They are providing a range of possible scenarios (deaths in this case) and a mean value. The range is typically mean +/- 1 standard deviation (not sure if it is exactly 1 SD in this case). The "mean" should not be confused with a "prediction".
Furthermore, the models are constantly adapting based on real time data. Due to the great number of unknowns and variables involved with a new virus, the range of possibilities is understandably large.

Guess this is like hurricane prediction maps with a 'cone of uncertainty'. They are predictive but with considerable uncertainty. Problem is, all people see are numbers with no cone of uncertainty. And most of that blame is on government officials and the press that use them for their own purposes.

 

[slykens]

The models are not giving a "prediction". They are providing a range of possible scenarios (deaths in this case) and a mean value. The range is typically mean +/- 1 standard deviation (not sure if it is exactly 1 SD in this case). The "mean" should not be confused with a "prediction".

I am in no way a statistician but it seems to me that at some point the model ought to come close to reality, otherwise what is the model modeling?

I'm not trying to be intentionally obtuse here but I'm trying to figure out what good a model is that doesn't even come close to what reality turns out to be. I'd accept GIGO as an excuse but that doesn't seem to be the case here, imo.

 

[interrobang]

Guess this is like hurricane prediction maps with a 'cone of uncertainty'. They are predictive but with considerable uncertainty. Problem is, all people see are numbers with no cone of uncertainty. And most of that blame is on government officials and the press that use them for their own purposes.

I equate them more to the winter storm model that shows one area getting 2 feet of snow 5 days beforehand, only for it to be flurries and 35 degrees 5 days later

 

[Grant Green]

I am in no way a statistician but it seems to me that at some point the model ought to come close to reality, otherwise what is the model modeling?

I'm not trying to be intentionally obtuse here but I'm trying to figure out what good a model is that doesn't even come close to what reality turns out to be. I'd accept GIGO as an excuse but that doesn't seem to be the case here, imo.

Hasn't been close to what? The mean value? Again, look at the high and low ranges.
From early April (I think assumes social distancing continued through summer):

Guess this is like hurricane prediction maps with a 'cone of uncertainty'. They are predictive but with considerable uncertainty. Problem is, all people see are numbers with no cone of uncertainty. And most of that blame is on government officials and the press that use them for their own purposes.

Yeah, kind of. The problem is that the general public doesn't look at the range, but just the mean, and assumes that is a prediction of exactly what will happen. However, the range is clearly shown on the model plots. I don't blame the public, as I haven't heard anyone in the media or government present models in the way they should be. I also don't think it helps that the UW model posts the mean at in the heading of the plot.

 

[OaktonDave]

Yeah, kind of. The problem is that the general public doesn't look at the range, but just the mean, and assumes that is a prediction of exactly what will happen. However, the range is clearly shown on the model plots. I don't blame the public, as I haven't heard anyone in the media or government present models in the way they should be. I also don't think it helps that the UW model posts the mean at in the heading of the plot.

The problem is that the model is not a useful tool if your trying to make policy decisions nor is it a good reference point for a general public that isn't well versed in modeling. From a policy standpoint, a range in excess of 100K is about as useful to policy makers as a forecast that says an hurricane is likely to strike the US somewhere between Galveston, TX and Portland, ME or an intelligence report that says some state or non-state actor in Eurasia will attempt to do harm to the US in the next 6 to 40 months. It just doesn't help make good decisions nor doesn't it really help novices understand what may happen. I have no doubt that the model has scientific value and understand that it was better than nothing in a situation that no one was really prepared for.

 

[Grant Green]

The problem is that the model is not a useful tool if your trying to make policy decisions nor is it a good reference point for a general public that isn't well versed in modeling. From a policy standpoint, a range in excess of 100K is about as useful to policy makers as a forecast that says an hurricane is likely to strike the US somewhere between Galveston, TX and Portland, ME or an intelligence report that says some state or non-state actor in Eurasia will attempt to do harm to the US in the next 6 to 40 months. It just doesn't help make good decisions nor doesn't it really help novices understand what may happen. I have no doubt that the model has scientific value and understand that it was better than nothing in a situation that no one was really prepared for.

Sure it does. If I'm trying to plan, I want to know if we are talking about 40-180k dead by August, or 2 to 4 million dead. Or what if the estimate was 10-30k? Perhaps we would have not gone into lockdown. Same thing goes for projections of hospital beds needed, vents, etc. It also helps to see model shifts so we can get an idea if actions like social distancing are having an impact.

 

[OaktonDave]

Sure it does. If I'm trying to plan, I want to know if we are talking about 40-180k dead by August, or 2 to 4 million dead. Or what if the estimate was 10-30k? Perhaps we would have not gone into lockdown. Same thing goes for projections of hospital beds needed, vents, etc. It also helps to see model shifts so we can get an idea if actions like social distancing are having an impact.

Sorry, we're going to agreed to disagree. I spent a career in risk management, and ranges that large don't offer enough to make a decision that's better than a WAG. From a public health perspective, we've lived with things that claim the lives in the lower range (and the stress on the health system that comes with such an event) and done nothing. The drastic actions needed to meet the higher know carry their own risks and impacts, as we all know too well. Telling a decision maker it could be catastrophic or manageable with normal actions just isn't that helpful.

 

[Grant Green]

Sorry, we're going to agreed to disagree. I spent a career in risk management, and ranges that large don't offer enough to make a decision that's better than a WAG. From a public health perspective, we've lived with things that claim the lives in the lower range (and the stress on the health system that comes with such an event) and done nothing. The drastic actions needed to meet the higher know carry their own risks and impacts, as we all know too well. Telling a decision maker it could be catastrophic or manageable with normal actions just isn't that helpful.

Ok, I guess the next time this happens we won't model it and we'll just guess at the number of people that could die. That would be a much better way to shape policy.

 

[UncleLar]

I'm not saying don't count and publish the antibody tests. But don't lump them in with the active case counts (that the media loves to run with to prove we're all still doomed) when they don't accurately portray active case counts.

The point that I was making is that the states track the data according to what they have determined are the needs of their particular state. They aren't acting irresponsibly. The concern is that the feds are accumulating data from disparate systems and lumping it together. That said, I would also argue that the Feds are acting responisibly in that they have purposely decided not to add any unnecessary reporting burden to the individual states in this time of need. If you want to point an irresponsibility finger, point it at the people (be it either the media or the general public) who don't understand and/or acknowledge that the data does combine the tests and are drawing false conclusions from it.

 

[OaktonDave]

Ok, I guess the next time this happens we won't model it and we'll just guess at the number of people that could die. That would be a much better way to shape policy.

Not at all. We can ask experts to use models and whatever other resources they have to form expert opinions and offer recommendations. That would give the models an appropriate place in the process and allow experts to bring other knowledge and experience into consideration when they are advising people who have to make real world decisions.

 

[Grant Green]

Not at all. We can ask experts to use models and whatever other resources they have to form expert opinions and offer recommendations. That would give the models an appropriate place in the process and allow experts to bring other knowledge and experience into consideration when they are advising people who have to make real world decisions.

How is that different than what has been done?

 

[OaktonDave]

How is that different than what has been done?

It would have been different because it would not have be widely disseminated in the public forum. The UW tool was/in useful to people who understood it, its value, and its limitations. It was and continues to be a source of confusion and misunderstanding for the general public who expected it to do something that it can't - predict the course of the virus with a considerable degree of accuracy. (If the model had met those expectations much of this thread wouldn't be here.) Journalists and public figures aren't likely to be able to explain it any better than the average person, and the ability of anyone with computer access to hit a link and see the model limits how much they would impact public perception if they could. The revelation that the UK's modeling tool was a bug filled mess that could not be salvaged only made matters worse.

I don't know to what degree it was used to inform policy makers, but its visibility suggests it was prominent. Given the wide variation in the projections, one can only hope that it was the best they could do in a difficult situation. I would be reluctant to present a technical tool to a non-technical audience because they audience is likely to respond much as the general public - not really understand that the dotted line representing the projection wasn't a true prediction. The alternative would be that they would look at it as I do, see the wide range in possible outcomes, and put the model aside while asking what the experts thought based on the all the information they had.

 

[Grant Green]

It would have been different because it would not have be widely disseminated in the public forum. The UW tool was/in useful to people who understood it, its value, and its limitations. It was and continues to be a source of confusion and misunderstanding for the general public who expected it to do something that it can't - predict the course of the virus with a considerable degree of accuracy. (If the model had met those expectations much of this thread wouldn't be here.) Journalists and public figures aren't likely to be able to explain it any better than the average person, and the ability of anyone with computer access to hit a link and see the model limits how much they would impact public perception if they could. The revelation that the UK's modeling tool was a bug filled mess that could not be salvaged only made matters worse.

I don't know to what degree it was used to inform policy makers, but its visibility suggests it was prominent. Given the wide variation in the projections, one can only hope that it was the best they could do in a difficult situation. I would be reluctant to present a technical tool to a non-technical audience because they audience is likely to respond much as the general public - not really understand that the dotted line representing the projection wasn't a true prediction. The alternative would be that they would look at it as I do, see the wide range in possible outcomes, and put the model aside while asking what the experts thought based on the all the information they had.

So, the model wasn't the problem. The problem was that it was presented to the public? Not sure how that supports your assertion that it wasn't useful for policy making.

 

[mn78psu83]

The new projections (updated June 10, 2020) are for October 1, 2020.

United States
169,890 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
13,177 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
32,310 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
7,164 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania

The June 8, 2020 projections for NJ, NY, PA added up to 52,502 deaths projected by August 4, 2020. The June 10, projections for NJ, NY, PA add up to 52,561 deaths projected by October 1, 2020. Interesting. By comparison, the June 10, 2020 data shows a projected increase of 24,162 total US deaths, again going out almost an additional 2 months.

http://www.healthdata.org/covid/updates
COVID-19: What’s New for June 25, 2020
Main updates on IHME COVID-19 predictions since June 15, 2020

GENERATING COVID-19 PROJECTIONS AND ALTERNATIVE SCENARIOS FOR ALL COUNTRIES
SARS-CoV-2 continues to spread globally, and for many locations, rising rates of COVID-19 infections, hospitalizations, and deaths are now occurring in the wake of eased or ended distancing policies. It is increasingly clear that COVID-19’s toll will extend beyond the summer months in the Northern Hemisphere, and current epidemics could easily worsen as the Southern Hemisphere enters its winter season.

United States
179,106 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
13,808 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
31,837 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
7,799 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania

 

[nits74]

I just listened to today's press briefing. All is well, we got this.

 

[Obliviax]

PA ranks 42nd in testing per capita. Who woudda thunk it?

https://coronavirus.jhu.edu/testing/states-comparison

 

[psualt]

http://www.healthdata.org/covid/updates
COVID-19: What’s New for June 25, 2020
Main updates on IHME COVID-19 predictions since June 15, 2020

GENERATING COVID-19 PROJECTIONS AND ALTERNATIVE SCENARIOS FOR ALL COUNTRIES
SARS-CoV-2 continues to spread globally, and for many locations, rising rates of COVID-19 infections, hospitalizations, and deaths are now occurring in the wake of eased or ended distancing policies. It is increasingly clear that COVID-19’s toll will extend beyond the summer months in the Northern Hemisphere, and current epidemics could easily worsen as the Southern Hemisphere enters its winter season.

United States
179,106 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
13,808 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
31,837 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
7,799 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania

The good news is 8 days ago in the US it was 201,129.

 

[Langmuir]

I would imagine modeling deaths for a novel virus is very difficult. You can start with some assumption of mortality rates, but what you don't know is if treatments are going to improve or not. I guess that is calculated into the uncertainty. Total cases should be something that is easier to model.

 

[mn78psu83]

The point of the COVID-19 models isn't to get the future number correct. It's to evaluate what change needs to happen to PREVENT that number from occuring. A change in the projected number isn't a sign that there is something wrong with the model. In actuality, it's the models that don't change that are the worthless ones.

@UncleLar This op-ed of June 24 in PNAS might be of interest to you.

https://www.pnas.org/content/early/2020/06/23/2011542117?cct=1971

 

[mn78psu83]

http://www.healthdata.org/covid/updates
COVID-19: What’s New for June 25, 2020
Main updates on IHME COVID-19 predictions since June 15, 2020

GENERATING COVID-19 PROJECTIONS AND ALTERNATIVE SCENARIOS FOR ALL COUNTRIES
SARS-CoV-2 continues to spread globally, and for many locations, rising rates of COVID-19 infections, hospitalizations, and deaths are now occurring in the wake of eased or ended distancing policies. It is increasingly clear that COVID-19’s toll will extend beyond the summer months in the Northern Hemisphere, and current epidemics could easily worsen as the Southern Hemisphere enters its winter season.

United States
179,106 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
13,808 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
31,837 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
7,799 COVID-19 deaths
projected by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania

http://www.healthdata.org/covid/updates
Last updated June 29, 2020 (Pacific Time)

United States
175,168 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
16,018 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
33,240 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
8,133 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania

 

[step.eng69]

@UncleLar This op-ed of June 24 in PNAS might be of interest to you.

https://www.pnas.org/content/early/2020/06/23/2011542117?cct=1971

Looks to be a good read, I'll read when I get home. In Kingston packing 50 yrs of my college books, office books, files, .Miscellaneous items. The owner of our small company is relocating me to a safer, less occupied lease location. Pretty much finished, over the last few weeks, packed about 25 boxes of books.

 

[PSU Mike]

Looks to be a good read, I'll read when I get home. In Kingston packing 50 yrs of my college books, office books, files, .Miscellaneous items. The owner of our small company is relocating me to a safer, less occupied lease location. Pretty much finished, over the last few weeks, packed about 25 boxes of books.

One of these things is not like the others ...

 

[UncleLar]

@UncleLar This op-ed of June 24 in PNAS might be of interest to you.

https://www.pnas.org/content/early/2020/06/23/2011542117?cct=1971

Thanks. One comment from the article resonates with me...

"Finally, “What will happen?'” is a different question than “What should we do?” "

The point that I've been trying to make is that the useful models are the ones that answer the latter question, not the ones that get the first answer most accurate.

 

[mn78psu83]

Thanks. One comment from the article resonates with me...

"Finally, “What will happen?'” is a different question than “What should we do?” "

The point that I've been trying to make is that the useful models are the ones that answer the latter question, not the ones that get the first answer most accurate.

Understood.

 

[mn78psu83]

Looks to be a good read, I'll read when I get home. In Kingston packing 50 yrs of my college books, office books, files, .Miscellaneous items. The owner of our small company is relocating me to a safer, less occupied lease location. Pretty much finished, over the last few weeks, packed about 25 boxes of books.

Yo WILDMAN!

 

[BobPSU92]

One of these things is not like the others ...

Or two of those things.

 

[mn78psu83]

http://www.healthdata.org/covid/updates
Last updated June 29, 2020 (Pacific Time)

United States
175,168 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
16,018 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
33,240 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
8,133 COVID-19 deaths
based on Current projection scenario by October 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania

http://www.healthdata.org/covid/updates
Last updated July 7, 2020 (Pacific Time)

United States
208,255 COVID-19 deaths
based on Current projection scenario by November 1, 2020
https://covid19.healthdata.org/united-states-of-america

New Jersey
16,970 COVID-19 deaths
based on Current projection scenario by November 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-jersey

New York
32,221 COVID-19 deaths
based on Current projection scenario by November 1, 2020
https://covid19.healthdata.org/united-states-of-america/new-york

Pennsylvania
9,999 COVID-19 deaths
based on Current projection scenario by November 1, 2020
https://covid19.healthdata.org/united-states-of-america/pennsylvania