By March 2020, there was a great deal of uncertainty about the impact that covid-19 would have. When Italy began to be heavily affected in mid-March, many European countries realized that covid-19 was likely to have significant consequences. To anticipate the evolution of the pandemic, policy-makers turned to the field of epidemiological modelling.

A team of scientists from Imperial College (London), led by Professor Neil Ferguson, published on March 16 the report that led many European governments, including the French government, to implement lockdowns. The scenarios presented in the report anticipated, without any measures, about 500,000 deaths in less than 3 months in the UK, and similar figures for France. Implementing standard measures such as planned in pandemic preparedness (mitigation strategy) plans would yield 250 000 deaths. The report argued that only the implementation of stricter measures to suppress the virus (such as lockdown) would allow the wave to be contained and that it would be the only way to avoid saturating the hospital system. Relying on this report, many governments have chosen to implement lockdown-style strategies.

Of course, projections of what would happen in the absence of measures cannot be compared with what actually happened in a country that implemented a lockdown. This is one of the most common counter-arguments to the critics of epidemiological models: it is precisely because of the measures that the modelling scenarios did not come true.

But there is one country that allows us to test the validity of the modeling scenarios: Sweden. In the first wave, Sweden chose not to confine itself, preferring to rely on lighter measures. It is therefore possible to compare what the scenarios for Sweden predicted and what actually happened.

The appendix of the Imperial College report 12, published on March 26, contains models of the epidemic in all countries, including Sweden.

The first 2 graphs represent the peak occupancy of ICU and hospital beds. The gray bars represent the Imperial College scenarios: early strict lockdown, late strict lockdown, and no action. The red bar represents the reality. Since Sweden has implemented social distancing measures without imposing lockdown, the observed numbers should have been somewhere between the “lockdown” and “no restrictions” scenarios. The reality is quite different. Even though Sweden did NOT implement lockdown, the scenarios WITH lockdown overestimate the peak of used beds by a factor 2 to 10. A scenario with some restrictions should be between lockdown and no restrictions scenarios, yielding a overestimation by a factor 10 to 60.

The last graph shows the different Imperial College scenarios for mortality in grey, and the actual number of deaths in red. Here again, the scenarios greatly overestimate mortality. The actual number of deaths corresponds to the early lockdown scenario, which was obviously not the approach taken by Sweden. The intermediate scenarios of social distancing overestimate mortality by a factor 3 to 7.

On April 29, 2020, during the first lockdown, an article in the newspaper Les Echos reported on unreleased models from the Pasteur Institute. These models were about the evolution of the epidemic post-lockdown. The Ministry of Health indicated that these models were not finalized and were being analyzed, and therefore could not be made public for the moment. To our knowledge, they have not been made public subsequently.

At the end of June, the most optimistic scenario overestimates the actual data by 66%. The median and pessimistic scenarios are 3.5 and 9 times above reality.

Although the report that led to these models has not, to our knowledge, been made public, they have nevertheless been communicated to the press and the government, as reported in the article in Les Echos:

It is perhaps the modelling carried out by the Pasteur Institute for the ARS Ile-de-France and the AP-HP - by Professor Simon Cauchemez - that invites the government to be cautious. Dated Tuesday and for internal use, they show that the number of patients in intensive care will remain high for a long time, at least until the summer. And that whatever the scenarios.

The confidentiality of the report from which these curves were taken raises the question of the transparency of the models. Similarly, the Scientific Council report of March 12, 2020 referred to the Imperial College models used in the report as “Neil Ferguson, personal communication”, without additional information.

As the models are not public, it was not possible for us to study the assumptions of the scenarios. The article in Les Echos simply mentions different reproduction rates for the virus, 1.2, 1.5, or more.

On October 26, 2020, Pasteur Institute produced scenarios for the evolution of the epidemic in the absence of lockdown. To our knowledge, this report has not been made public, and we have not been able to consult it. We simply found a figure extracted from the report, which can be found on this page.

On October 28, 2020, faced with the prospect presented by these figures, Emmanuel Macron announced a generalized lockdown, which would take effect on October 30.

On October 30, the Institut Pasteur produced an update of its scenarios to take into account the impact of lockdown. The report has not been made public to our knowledge. However, some media have had access to it, and have reproduced the scenarios. The article from Les Echos is our source for our graphics.

The most optimistic scenario is close to the actual peak of occupied ICU beds (+15%). The median and most pessimistic scenarios are at +30% and +100%.

During the emergence of the British alpha variant, the Scientific Advisory Board report of January 12, 2021 has produced initial preliminary estimates of its highly uncertain impact, both in terms of the size of the hospital peak (from 7,000 to 40,000 critical care beds) and the date of the increase (from February to April), making it impossible to compare the curves with reality.

However, these projections will be refined in a succession of reports from INSERM and Institut Pasteur from mid-January to mid-February. These scenarios were used as a basis for considering the implementation of strict containment in early February, which in the end did not take place, providing an opportunity to compare the projections with what actually happened.

From December 15, 2020, an 8pm curfew was in place throughout France. It was be brought forward to 6pm for 15 departments on January 2, 2021, then for the whole country on January 16.

The first regional lockdowns were implemented on March 20, followed later by others until the national lockdown on April 3. The latency between the implementation of measures and their effect on hospitalizations being at least one week, and even 2 to 3 weeks as indicated by the scientific council report (the first effects of the measures adopted […] can only appear after two to three weeks), we stop our graph at March 27. Beyond that point, it is no longer legitimate to compare the dynamics of the pandemic to the no lockdown scenarios.

Note: the January 16 and February 2 scenarios partially considers the impact of the 6pm curfew. The February 14 scenario fully considers them. We discuss the choice of scenario inclusion below in the Discussion of Scenario Assumptions section.

The INSERM scenarios (here, here and here) carried out over 1 month all point to an exponential increase in hospital admissions. The reality will be a slight decrease followed by an increase back to around the initial level. Virtually all scenarios exceed 30,000 weekly admissions by mid-March, with no sign of a slowdown in momentum, whereas the reality is around 10,000, or 3 times less. The most optimistic scenarios overestimate hospital admissions by a factor of 50-100%.

Note: all these scenarios take into account the impact of the 6pm curfew.

For the February 8 report, the 2 main scenarios extending into March reflect a “no vaccination” hypothesis and an “optimistic” vaccination hypothesis. It is specified that ” in practice, the impact of the current campaign is likely to be intermediate between the scenarios with and without vaccination”, hence our choice to represent these 2 curves. Later in the report, secondary variants are presented to illustrate the influence of a small variation in reproduction rate: these are the curves stopping at the beginning of March.

The most optimistic scenario sticks to reality until mid-March. This was not the main scenario presented to decision-makers (cf. the 2 scenarios extending into March). The others overestimate actual admissions by a factor of 50-100%.

For the February 23 report, the median scenario broadly corresponds to reality, which is around the median scenario.

No mention is made of the significant difference with the scenarios in the previous report of February 8.

A retrospective assessment of the February 8 report will be made in a subsequent April 26 report. It states that

As early as February 8, this model anticipated that the curfew would be effective in containing the circulation of the historical virus but would not prevent a significant increase in hospitalizations in the absence of additional measures.

This statement is accompanied by the figure below. One year into the pandemic, this figure will be the first retrospective comparison officially published by the Pasteur Institute.

However, we could not find such a projection in the February 8 report. All those we found are represented on our graphs presented above.

The INSERM modelling of January 16 explicitly advocates for thougher social distancing measures in its summary :

“These findings show the need to strengthen social distancing measures”

Strikingly, it acknowledges not taking into account the curfew, a measure whose very target was not to implement a lockdown

It does not account for the curfew anticipated to 6pm and extended to the national territory on January 16, 2021. The evaluation of this social distancing measure will be performed on epidemic data in the next few weeks. For this reason, we provided a more optimistic scenario assuming a stable epidemic activity.

Note that while the impact of the curfew is only partially taken into account in the reports of January 16 and February 2, it is fully integrated in the report of February 14. You will find more details in the following section Discussion of (technical) scenario assumptions.

These results were also presented in a Scientific Council Special Report on January 29, 2021, stating that these results show the need for rigorous social distancing measures to deal with the threat of the variant and studying the implementation of strict containment in early February.

About the INSERM scenarios:

• January 16 report: the scenarios correspond to 3 reproduction rates of the historical variant (1, 1.1 and 1.2), coupled with a 50% or 70% more transmissible alpha variant. The curves are taken from Figure 1.

• report of February 2: the scenarios correspond to 3 reproduction rates of the historical variant (0.9, 1.1 and 1.2) coupled with a more transmissible alpha variant 50%. The curves are the gray curves in Figure 2 (redundant with the curves in Figures 1 and 3).

• February 14 report: 50% more transmissible alpha variant. The transmissibility of the historical virus is not given. In addition to the standard projection, 2 scenarios exploring a variation in the transmission of the historical variant of +/- 10% illustrate a reduction or a reinforcement of the measures. The reality did not correspond to either of these 2 scenarios, but we chose to represent these 2 scenarios so as not to be accused of arbitrary exclusion, since a 10% variation could naturally and randomly occur.

About the Pasteur Institute scenarios:

• report of February 8: the scenarios correspond to 3 reproduction rates of the historical variant (0.9, 0.95 and 1), coupled with a 40%, 50% or 70% more transmissible alpha variant. The curves are taken from figures 2a, 6a, 7c

• report of February 23: many variations are proposed, because as the report underlines in view of the great uncertainty “a single forecast for the coming months is not possible”. In detail, the different hypotheses concern the greater transmissibility of the alpha variant (+40%, 50% or 70%) and a change in the epidemic trajectory (decrease and increase of 8% or 16%) which occurs at different dates (February 22 or March 8). The curves are extracted from figures 2c, 5a, 5c, 5e, stopping at 27 March before the lockdown as explained in the “context” paragraph.

About all scenarios

The alpha variant is assumed to be only more transmissible, and does not increase the risk of hospitalization. This will be modified in subsequent reports, for example the one from April 26, 2021, where the probability of hospitalization is increased by 64% with alpha infection.

For all reports, we stop extracting curves on March 27 as explained in the context paragraph, the first confinements being implemented on March 20 and the effect of the measures on hospitalizations taking at least a week to materialize (cf scientific council report, “the first effects of the measures adopted […] can only appear after two to three weeks”).

About scenarios that do not include the anticipated curfew.

The INSERM reports of January 16 and February 2 deserve clarification, as they state that the effect of the early 6 p.m. curfew is not taken into account, but that the most optimistic scenario could reflect the measure. Should we have kept only this one and excluded the other scenarios? Our decision to keep them all was based on several points:

• First of all, the authors have chosen to represent them, indicating that they themselves validate the relevance** of these scenarios. Indeed, it is on the basis of these scenarios that they assert that “new hospitalizations are expected to reach around 25,000 between mid-February and early April in the absence of intervention”.

• Based on these results, they also gave explicit policy recommendations**: “These results show the need to reinforce social distancing measures”, even though the curfew, whose very purpose is to avoid more restrictive measures, was not modeled.

• Finally, we see that the scenarios for January 16 and February 2 are very similar to those for February 14, even though the latter incorporates the effect of the anticipated curfew, indicating that taking the curfew into account changes the results only slightly.

Scenarios are published on April 26 and May 21, covering epidemic trajectories for the end of the 3rd containment period and the summer. We stop the comparisons in mid-June, when the delta variant appears, and the models cannot anticipate this.

An informal self-evaluation of the May 21 report was published on the twitter account of Simon Cauchemez, in charge of the Institut Pasteur modeling team. To our knowledge, this was not done for the April 26 report. The self-assessment does not mention the difference observed with the April 26 scenarios.

For the first report, the scenarios are above reality in mid-June, with overestimation ranging from a factor 2 to 6.

On the other hand, the observed recession follows faithfully the scenario of the second report, as shown in the screenshot of Simon Cauchemez. He does not mention the discrepancy in the first report.

A first report from the Pasteur Institute came out on July 9, 2021 in the context of the delta variant spread. Two out of three scenarios presented anticipated a number of critical care beds by the end of August at least equal to the 2nd wave, and in the absence of measures a peak in September well above the 1st wave.

Faced with the prospect of hospital flooding suggested by these scenarios, the extended sanitary pass was implemented on 21 July, leading to a sharp increase in vaccinations and rendering the hypotheses of the 9 of July report obsolete.

To remedy this, the Institut Pasteur published two new reports, on July 26 and August 5, to take into account the effect of the sanitary pass. We compare these two reports with the actual trajectory.

For the first report, the reality is about 2 times smaller than the most optimistic scenarios, 3 to 4 times smaller than the median scenarios, 10 times smaller than the pessimistic scenarios.

The report published 10 days later partially corrects this overestimation, the reality corresponding approximately to the most optimistic scenario, but remaining 2 times smaller than the median scenario and 4 times smaller than the pessimistic scenario.

In July, the United Kingdom faced a similar situation to France: an increase in hospital admissions, accompanied by scenarios anticipating hospital flooding. Unlike France, the UK did not implement a sanitary pass and lifted almost all control measures.

We present a comparison between modeling scenarios and reality in the case of “Freedom Day”, the day on which almost all restrictive measures were lifted in the UK. On July 19, as part of its exit plan, the British government decided to lift most of its restrictive measures (such as capacity limitations in places open to the public, compulsory wearing of masks or travel restrictions).

The data presented below compares actual hospitalized patients with different modeling scenarios to predict the impact of lifting restrictions. The data are from the July 7 SAGE (Scientific Advisory Group for Emergencies) report, and the formatting is from The Spectator.

The actual peak is below the most optimistic scenario, the median scenarios are 2-3 times higher, and the most pessimistic scenario is more than 5 times higher.

This lifting of restrictions was strongly criticized by many scientists in an open letter to the prestigious medical journal The Lancet. They called it “dangerous and premature”. This open letter was picked up by various British and international media outlets, which relied on these models to announce “a third major wave of hospitalizations and deaths” following the lifting of the restrictions, described as a dangerous and unethical experiment”.

The fact that the models greatly overestimated the impact of the reopening was described in the scientific journal Nature by a renowned epidemiologist:

No one really understands what’s going on.

In a context marked by the generalization of the sanitary pass, the Pasteur Institute published models on the epidemic dynamic during the autumn-winter transition. Although the arrival of the Omicron variant made these models obsolete in December, there is still a 2-month period over which they can be compared with reality.

Most scenarios are at the level of a contained epidemic circulation, below actual hospital admissions by a factor of 4 to 6.

Scenarios closer to reality correspond to a “relaxation of measures/behaviors by 40%”, which returns to the “control level measured in June-July”, coupled with an unfavorable climate modulation. Although the control measures were not relaxed to the June-July level, when there was no sanitary pass, it is impossible for us to measure the “relaxation of behaviors”, so we kept these scenarios.

These models were taken up by the Scientific Council in its report of 5 October 2021. It recommended that the sanitary pass be maintained for a period running at least until November 15, 2021 (first extension of the sanitary pass). They were also reported in the national press.

Our curves are taken from figures 7 and 9.

• Behaviors/measures reducing transmission to the level of June/July before the sanitary pass (-40% compared with June/July), or to an intermediate level (-20%).

• Climate modulation increasing contamination by 33% (estimated), 20% or 40%.

• A relaxation of measures/behaviors on October 15 or November 15. The dates of December 15 and January 15 were also studied, but are not of interest to us as they fall outside our study period.

• 3 hypothesis variants for vaccine efficacy against hospitalization and infection: 95% and 60% (reference), 90% and 60%, and 95% and 80% respectively.

• Vaccine efficacy against transmission of 50% if infected.

• The delta variant increases the risk of hospitalization by 50% compared to the alpha variant.

• Alpha variant increases the risk of hospitalization by 42% compared with the historical strain.

As said before, the “relaxation of social distancing” is an hypothesis impossible to verify. An imperfect measure of it would be Google mobility trend, which shows no real increase over the period :

In the context of a second delta wave, Pasteur Institute produced 2 reports on November 29 and December 2, 2021 with scenarios covering winter and spring 2022. Unfortunately, the emergence of the omicron variant 2 weeks later renders the hypotheses obsolete and does not allow a comparison with reality.

A criticism often made of the comparison of real data to “scenarios” as performed above is that the latter are not “predictions”, and simply serve to give an idea of the major trends and impact of different measures.

Nevertheless, beyond the scenarios presented in its reports, the Institut Pasteur also publishes “Short-term projections of hospital needs” on this page , which are regularly updated. Here, the goal is indeed to correctly forecast the number of patients, the Institute stating that “the relative error for projections of the number of critical care beds at the national level is 11% at 14 days.”

Here we compare the model projection 2 weeks before the peak of the second delta wave in mid-December 2021. This peak occurred just before the emergence of the omicron variant, which changed the game in the weeks that followed.

At the time of the peak of the delta variant, the model projection corresponds to an overestimation 2 weeks later of 30% for critical care beds, and of about 40% for the other indicators.

Despite an average model accuracy of 10%, at the time of the peak the actual error will have been much higher. We would have liked to compare the model projections to the other peaks (February 2021, April 2021, August 2021) in order to assess whether this was a one-time error or a systematic bias on the peaks. Such an evaluation is important because it is the anticipation of a hospital peak that conditions most of the policy response.

Unfortunately, the regular updating of projections on the page presenting the results overwrites previously published projections, and we were unable to make this systematic assessment. Keeping previous results in open data would help such an approach and an appropriation of the results by citizens.

Following the appearance of the Omicron variant, the Pasteur Institute published a first report on December 27, 2021, presenting more than a hundred scenarios. Indeed, it is specified that “given the uncertainties […], it is not possible to quantify precisely the impact that the Omicron wave will have”. Therefore, we did not analyze this report.

The following report of January 7 takes as a basis the previous one in the light of the new data, and excludes the scenarios that do not fit the Omicron properties, making it possible to analyze the results.

We also analyze a report published on February 15, which is a self-assessment comparing what was “predicted” (the term used by the report) with what actually happened.

The report distinguished between “likely” scenarios (solid lines) and “less likely” scenarios (dotted lines). It also distinguished between a hypothesis of 85% or 60% efficacy of the booster dose against infection at 6 months.

The reality is an initial efficacy of 50%, then 0% beyond 3 months (see paragraph “Institut Pasteur’s self-assessment”), excluding at the very least the 85% scenario. However, as the Institut Pasteur uses it to assert in its retrospective report that “the predicted and observed dynamics of daily admissions are close”, we have chosen to represent it.

The peak in critical care beds will occur a few days after publication of the report, instead of the month and a half anticipated. The height of the peak will be slightly below the most optimistic scenario, but more than 2 times less than the median and pessimistic scenarios. The “unlikely” scenarios further overestimate reality.