COVID-19 Risk Assessment Simulation Model

Rebekkah Ismakov
December 19, 2020
11 min read

Motivation

Carbon Health’s COVID Ready program is helping businesses across the country get their employees back to work safely. As a first step, our data team has built a simulation model that mimics transmission in the workplace and quantifies the number of cumulative infections over a given period of time. We can then use this simulation to tune parameters, taking into account certain constraints that the business may have — budget or otherwise — to determine the best policies and protocols for businesses looking to get their employees back to work safely.

Not only will this simulation allow us to assess how efficient we would be at stopping outbreaks within the workplace, but it will also allow us to inform certain discrete actions for those businesses. These actions include decisions around which type of diagnostic tests to use, at which cadence to test employees, what quarantine practices to implement, and how to best prioritize who to test.

The Compartmental Model

We created an agent-based SIR model that simulates the actions and dynamics of individual agents and assesses their effects on the overall system. These individual agents progress between S, I, R (Susceptible, Infectious, Recovered) compartments in the model depending on their interactions with the population. One of the benefits of agent-based models is their ability to model highly complex systems. This model is capable of incorporating a level of randomness while simulating individual actions and interactions, and then tracking the pattern on the whole aggregate system as the data is generated from the simulation. The main framework for this model is from a post investigating the effects of different types of COVID-19 diagnostic tests. [1]

The parameters used in the simulation are:

  • I_initial: the initial number of infected employees.
  • testing_interval: the frequency in which tests are administered.
  • testing_delay: the turnaround time from administration of test to delivery of results.
  • sensitivity: the test sensitivity (i.e. the true positive rate).
  • beta: the likelihood of workplace infection.
  • external_infection_rate: the likelihood of infection outside of the workplace.
  • gamma: the recovery rate.
  • Q_duration: the duration period of quarantine.
  • N: the total number of employees in the workplace.
  • risk_behavior: the likelihood that an employee would choose to remain home given COVID-19-like symptoms (including both actual COVID-19 symptoms as well as COVID-19-like symptoms from other illnesses).
  • testing_process: the prioritization method (‘random’ , ’symptomatic_first’, ‘asymptomatic_first’).
  • sym_pos_ratio: the proportion of infected employees who are symptomatic.
  • sym_neg_ratio: the proportion of non-infected employees who are symptomatic (due to flu or other non-COVID-19 respiratory illnesses).
  • num_days: the number of days to simulate.

These parameters can all be adjusted for each specific business. For example, depending on the workplace policies on masks, on distancing, and on sanitation, the `beta` value may increase or decrease. The industry sector of the business will also affect these values — a meatpacking plant will likely have a higher `beta` value than would a tech company. Similarly, the `external_infection_rate` is highly dependent on both location and employee behavior.

We start with an `N` number of employees. We then take the following steps:

  1. Introduce initial infections where a certain number of employees (`I_initial`) out of `N` total employees contract the SARS-CoV-2 virus.
  2. Introduce symptoms in which a certain percentage of cases develop symptoms (depending on `sym_pos_ratio` and `sym_neg_ratio`). These include both COVID-19 symptoms, as well as COVID-19-like symptoms (due to, for example, the flu or other respiratory illnesses with similar types of symptoms).
  3. Introduce self-imposed quarantine where a certain percentage of cases displaying symptoms choose to self-isolate, depending on the `risk_behavior` parameter. Ideally, we would hope that all individuals with symptoms self-isolate, but realistically we know that due to misdiagnosis of symptoms (i.e. associating a cough with allergies) or inability to isolate for a variety of reasons, this will not be 100%.
  4. Run tests on selected cases where selected cases (depending on `testing_process`) take COVID-19 tests.
  5. Deliver test results where patients receive their test results and act accordingly, that is, the patient will quarantine if positive. The accuracy of the test results will depend on the `sensitivity` parameter.
  6. Recover number of infected cases, infect number of susceptible cases. Depending on the `gamma` (recovery rate) parameter, some cases will recover. Others will become infected.
  7. Repeat steps 2–7. Stop when the count is equal to `num_days`, indicating the end of the simulation.

The output of the simulation on each simulated day is 1) `Q`, the number of employees quarantining, and 2) `I`, is the number of (non-quarantining) infected employees. We then calculate the probability of contracting COVID-19 for each employee using the following formula:

Image for post

In short, the probability of contracting COVID-19 is one subtracted by the probability of coming in contact with only safe, non-infectious, employees.

Testing Priority

Note that it would be ideal to test everybody, but due to limitations in the number of tests or constraints in budget allocation, this might not be a possibility. In order to decide who to test, we included a few filters to Step #4 above. We prioritize patients based on the following heuristics:

  1. Symptom-based: This depends on how the simulation decides to prioritize patients with options to test symptomatic first, asymptomatic first, or random selection (i.e. not based on symptoms).
  2. Recency-based: This depends on the number of tests available. We’ll order the employees by days since their last test, and prioritize those who have been tested less recently.
  3. Immunity-based: If a patient tested positive, quarantined, and then tested negative, for this simulation we assume that individual won’t contract COVID-19 again*. (*It should be noted, however, that recent studies have found that cases may re-contract COVID-19 again and that immunity to the virus is not a given).

If additional tests remain after all the filters are applied, we randomly select who to test out of the remaining employees.

Testing Sensitivity vs Turnaround Times

For COVID-19 testing, we currently use two different types of tests:

  • PCR tests, which look for the genetic material of the SARS-CoV-2 virus to determine if an individual has an active infection. PCR tests include both rapid and non-rapid versions.
  • Antigen tests, which look for specific viral proteins that make up the SARS-CoV-2 virus to determine if an individual has an active infection.

The PCR non-rapid tests are believed to provide greater testing sensitivity (i.e. more reliable), but take longer to deliver results. The greater the number of tests administered, the longer this delay will be. This is because a large influx of tests leads to bottlenecks in the laboratory process. By contrast, antigen tests and PCR rapid tests have much quicker turnaround times — results within 15 minutes — but are believed to be less reliable, displaying a higher number of false negatives.

So which is preferable of the two? Do we prefer greater sensitivity at the expense of longer turnaround times, or vice versa?

Outcomes

Experimental Setup

As a first step, we run the simulation with the following default values:

  • I_initial: 5 cases
  • testing_interval: 4 days
  • beta (likelihood of intra-workplace infection): 0.2
  • external_infection_rate: 0.001
  • gamma (recovery rate): 0.07
  • Q_duration (quarantine duration): 14 days
  • N: 200
  • risk_behavior: 0.0
  • testing_process: random
  • sym_pos_ratio: 0.65
  • sym_neg_ratio: 0.05
  • num_days: 130

For the antigen and rapid PCR tests, we use a test sensitivity value of 0.75 and a testing delay value of 0 days (since results are returned within 15 minutes), while for the non-rapid PCR test, we use a testing sensitivity value of 0.98 and a testing delay value of 5 days. It should be noted that in reality, this testing delay is often even longer, up to 9 days long. Additionally, while the exact testing sensitivity is out of the scope of this study, this model will allow us to investigate the relative benefit of faster turnaround times compared to lower sensitivity.

We ran the model through 100 iterations for each configuration type, with an iteration meaning a full run of the 130-day simulation, and then observed the average number of cumulative infections for each day over a given period of time. This allowed us to plot and compare the different parameter configurations to determine the configuration leading to less cumulative infections.

Simulation Results

First we compared the impact of faster turnaround times versus greater sensitivity. As shown in the plot below, faster turnaround times result in lower numbers of infections even at the expense of lower reliability. This insight is valuable and non-intuitive. It is important to consider this tradeoff when determining the best diagnostic tool.

We also plotted the best case scenario of zero intra-workplace infections, where a slight increase in transmission rate is visible due to purely external infections that have occurred outside the workplace.

Image for post

However, in the above plot we were assuming a `risk_behavior` of 0, meaning that all simulated employees were choosing to head to the office regardless of whether they displayed symptoms or not. Hopefully this is not the case — individuals with symptoms are likely to self-isolate in most cases.

We then decided to investigate how transmission rate will be affected by different `risk_behavior` values. Testing values of 0.0, 0.25, 0.5, 0.75, and 1.0, results showed that even low `risk_behavior` values greatly decreased outbreaks. When `risk_behavior` was just 0.25 (indicating that 25% of cases with symptoms will self-isolate), cumulative infections dropped from 31.20% to just 17.08% of total employees infected.

Note that here we are using the `sensitivity` and `testing_delay` values of the antigen test, so the black line (‘0% of those with symptoms remain home’) in the below plot corresponds to the blue line ( ‘Antigen test every 4 days’) from the previous plot.

Image for post

We then tuned the `risk_behavior` parameter and tested antigen vs PCR testing again. This time, we used a `risk_behavior` value of 0.4 (meaning 40% of cases displaying symptoms will choose to self-isolate) and a value of 0.9 (90% of cases displaying symptoms choosing to self-isolate).

Image for post
Image for post

Even when 40% of those with symptoms were choosing to remain home, antigen tests were outperforming PCR non-rapid tests in diminishing transmission. When the value goes up to 90%, the difference between the tests diminishes, but antigen tests still outperform PCR tests, showing slightly fewer cumulative infections.

Keep in mind here that the y-axis scales differ between the two plots above. We see a higher number of infections with the configuration where `risk_behavior` = 0.4 than we do with the configuration where `risk_behavior` = 0.9. Although antigen tests result in less cumulative infections than do PCR tests in both cases, PCR testing with 90% of employees self-isolating with symptoms yields less infections than does antigen testing with only 40% of employees self-isolating with symptoms. This further showcases the importance of employees choosing to remain home with the onset of symptoms.

The next parameter we experimented with was the `testing_cadence` values. We kept the default antigen test values for the `testing_delay` and `sensitivity` parameters, and then observed how transmission rate was affected when we tested employees every 2 weeks, every week, every 4 days, 3 days, 2 days, everyday. Not surprisingly, the more frequent the testing, the lower the transmission rate.

Image for post

It should be noted that in all of the above cases, we were randomly selecting who to test. Given that people with COVID-19 are more likely than not to display symptoms, we next observed how the testing prioritization criteria affects transmission. Instead of randomly selecting who to test, we observed how outbreaks would be affected by prioritizing symptomatic or asymptomatic cases first.

Image for post
Image for post

As seen above (top), when all employees are coming into the workplace, it makes sense to first test symptomatic cases. Does this change if more people are choosing to remain isolated when displaying symptoms? Yes, the difference diminishes when the `risk_behavior` value is changed to 0.9. If people with symptoms remain home, the testing prioritization method is less relevant.

Conclusion

In summary, the measures that can be taken to prevent COVID-19 outbreaks in the workplace include:

  • Choosing diagnostic tests with faster turnarounds times even at the expense of decreased testing sensitivity.
  • Increasing testing cadence for employees.
  • Having individuals who are displaying symptoms self-isolate.
  • When employees are unable to self-isolate with symptoms, prioritize testing for symptomatic employees first.

This current configurable model is the framework for organizations looking to assess relative risk depending on certain parameters. However, figuring out the specific values to use within the model is not a straightforward task. For the above analyses, we used a value of 0.2 for the `beta` parameter. This specific value will depend on a myriad of factors though, and will differ among different work sectors — a meatpacking plant will likely have a higher `beta` value than would a tech company. Policies on wearing masks, protocols around sanitization and ability to social distance in the workplace will affect this value. Vaccinations will decrease the value of this parameter as well — the amount by which depends both on the number of employees vaccinated, as well as the efficacy of the vaccines.

Similarly, the `external_infection_rate` will depend on variables such as the individuals’ locations and behaviors (i.e. do they take public transit regularly). To reliably calculate these values, we will further need to look at current literature on the subject as well as our Carbon Health dataset. We are prioritizing this research to make sure we are able to provide a reliable model, with up-to-date information.

Next steps for this project include creating a web application. This would allow an organization to input certain information and constraints, including budget, number of employees, sanitization practices, work from home and sick leave policies, distance between employees, etc. The application would then recommend testing cadence and testing prioritization strategies for that particular organization.

Although it is difficult to reliably gauge the exact values for certain parameters (for example, it is difficult to know the exact value for the `beta` parameter), this can still be used to assess relative risk and to compare two different policies (for e.g. testing prioritization strategies).

Ensuring workplaces and businesses are opening safely is important. With more data and careful analysis, we are able to better select between testing options and better inform] employees to self-isolate, thereby reducing or diminishing outbreaks.

If you have any suggestions or questions regarding this study, please reach us at data-science@carbonhealth.com.

Acknowledgments

Thanks to my colleagues at Carbon Health for their valuable feedback on this study:

Data Science Team: erenbali, Raymond Deng, Neha Kumar, Nosheen Moosvi, Hoda Noorian, Pardis Noorzad

Medical Team: Caesar Djavaherian, MD, Sujal Mandavia, Aaron Weinberg, Roger Wu

Clinical Informatics: Haritha Atluri

Marketing Team: Caitlin Heinz, Loren Mattia

References

[1] https://chris-said.io/2020/07/15/why-inaccurate-antigen-tests-are-better-than-slow-pcr-tests/