By: Katie Speanburg, Laura Peck, Lauren Olsho, Christopher Spera, and Alvaro Cortes

Introduction

The World Health Organization’s (WHO) designation of COVID-19 as a pandemic will likely require health, social, and economic researchers to reconsider their study designs for ongoing research. This brief outlines how the COVID-19 pandemic could affect the design phase, data collection, and the analysis approach of health, social, and economicresearch.

Considerations for the Design Phase

Study teams are still learning about the effect of COVID-19. They should consider modifying their study design to improve theworld’s ability to respond to COVID-19 and learn about the pandemic’s potential effects on an array ofoutcomes. And of course, studies that require in-person interactions should carefully consider the health and well-being of research staff and study participants and determine if another data- collection method is appropriate.

(1) Improving the world’s ability to respond. A few options for consideration include:

• Adding questions to surveys, focus groups, and interviews about people’s attitudes toward social distancing and how people are (or are not) mitigating the spread of the virus
• Adding new data collection activities that shed light on the transmission of the virus and treatment options
• Adjusting the timeline for implementing an intervention if the interventionis designed to enhance the capacity of study participants to successfully enter thehealthcare industry. For example, in a study of participants in healthcare sector training—such as becoming Registered Nurses—the training/service providersmight accelerate their efforts(rather than shut down) to prepare trainingparticipants to join thehealthcare workforce quickly and respond toCOVID-19 sooner.

(2) Understanding the pandemic’s potential effects on an array of outcomes. While COVID-19’s impact on people’s health is the top concern, we also recognize that COVID-19 has affected a broad array of other important outcomes—e.g., employment, financial stress, educational attainment, substance use disorders, and homelessness (to name just a few). As a result, study teams could consider the effect of COVID-19 and rethink the outcomes of interest in their studies, which, in many cases, can be explored using administrative data. For example, study teams can leverage these administrative data:

• Unemployment Insurance
• National Directory of New Hires
• Social Security disability claims
• Census tracts
• Consumer Financial Protection Bureau data
• Medicaid or Medicare enrollment data and claims
• Local Homeless Management Information System data
• National Center for Education Statistics data.

If there are additional data sets that should be analyzed, then researchers should plan ahead to collect those data within the appropriate timeframe.

(3) Health and wellbeing of research staff and study participants. The pandemic obviously affects research staff and study participants. The design plan should include a clearly defined contingency plan should research staff become unavailable.Study teams must be prepared to make key design decisions in the potential absence of key technical staff. Operationally, study teams will encounter disruptions during the lifecycle of a study, especially since many organizations have moved to telework. That can cause confusion about review and approval processes and staff availability as staff juggles work and personal responsibilities. Accordingly, it is important to have staff who are cross-trained in many roles and can quickly pivot as needed. And study managers must adopt proven project management approaches to maintain appropriate staffing and deliver high-quality work. Design plans should also include well-conceived protections for human subjects that an Institutional Review Board (IRB) has approved.

Considerations for Data Collection

COVID-19 is likely to affect a study’s data collectionactivities considerably. For example, organizationsmay decide to suspend all in-person datacollection activities to prevent the spread of the virus. The intended study population also may be temporarily unavailable during the pandemic. They may include:

• Healthcare professionals who focus on treating patients
• Hospital or nursing home patients who can no longer accept visitors
• School-aged children who are not in the classroom
• Homeless populations in shelters
• Individuals who are incarcerated and unavailable to participate in research during this time.

Study teams have several options:

(1) Consider switching modes from in-person to virtual data collection. With many options to reach people virtually, study teams are able to adapt and switch data collection modes. However, the switch may introduce other types of challenges to data quality, integrity, and compliance within the boundaries of applicable Federal regulations. For example, study teams may need to train data collectors on the virtual data collection tool, and once data collection begins, teams must confirm that the correct respondent was reached. Also, study participants may not complete the entire survey or interview due to challenges with internet connectivity. Lastly, virtual data collection complicates the study team’s approach to obtaining consent and complying with the data security provisions of the Health Insurance Portability and Accountability Act (HIPAA).

(2) Consider discontinuing data collection early if the study has sufficient data and statistical power to answer the research questions. In some situations, study teams may be able to suspend their data collection activities if, in conjunction with relevant stakeholders (e.g., federal government contract officer), the study team determines that it has sufficient data to meet the objectives of the study.

(3) Consider pausing data collection until the population becomes available if doing so would not compromise the research. Temporarily suspending data collection is likely to be a common situation, especially when study participants are inaccessible. For example, consider research with children at this time. If children were in a classroom receiving an educational intervention before COVID-19, it is possible that the intervention will be found to be ineffective after COVID-19. However, this conclusion could be misinformed if there was a several month gap when the school was closed, or if the intervention was conducted in a virtual classroom where it could not be implemented effectively. When data collection is resumed, researchers will need to consider the impact on the quality of the data and bias in responses, post-pandemic. (This is discussed in the section below.)

(4) Consider adding another data collection point to learn more about potential impacts of COVID-19, depending on the purpose and topic of the research. This additional data point could be relevant now and for several years following the end of the pandemic. A cautionary note, however—this should not be a fishing expedition! Once the study design is finalized, incorporating COVID-19 considerations as appropriate, pause to reflect on the decisions. If those decisions are still relevant given the daily changes in our environment, then proceed with the research plan. Project leadership will need to regularly reassess the study design to ensure it is meeting relevant Federal, state, and local guidance and mandates as well as ethical considerations.

Considerations for Analysis Phase

When conducting analyses for a project that collected data during the pandemic, it will be important to look at the interplay of statistical power and respondent biases. It may be the case that the study gets a higher response rate than anticipated because many people are working from home and are therefore potentially more likely to answer an email or phone call from an unknown entity. Better survey response rates and larger sample sizes will improve the statistical power of the study to detect impacts. Depending on the topic of the research, however, responses are likely to be affected by the pandemic, which could introduce bias into the results.

As research and evaluation professionals, we are always taught to maximize internal validity and guard for threats to internal validity, such as history. This is definedas an external event that that occurs while the study is in progress that could skew the findings, provide an alternative explanation for the intervention having a null or significant effect other than the intervention itself, and/or muddle the interpretation of findings when ‘pooling’ impact estimates. COVID-19 certainly poses the kind of threat to internal validity that could have wide- ranging impacts to ongoing research.

An event of history impacting internal validity in the last 20 years was 9/11 (September 11, 2001). Many studies that were in the middle of data collection showed significant differences pre- and post-9/11. Although an external circumstance like COVID-19 should theoretically affect participants in the treatment and comparison groups equally, studies back then found thatadditional work was neededwhen calculating impact estimates related to an intervention before and after 9/11.

The degreeof bias may depend on the extentto which evaluation sites in treatment and comparison groups are differentially located in COVID-19 hotspots. They may be amoving target given the many unknowns about the epidemiological trajectory in different regions ofthe country. In experimental evaluations, we assume that randomization enables the treatment and control group to account for all differences except the intervention. However, since COVID-19 heavily targets some communities and not others, the impact estimates, and resulting bias, may not be minimized bythe nature of the experimental design. In quasi-experimental design studies, the potential biases may be even more acute. In addition, there are important questions about whether the magnitude of the intervention’s effect changes (increases or decreases) pre- and post-COVID 19.

To handle these issues, study teams may revise their analysis plans to:

(1) Conduct subgroup analyses. For example, researchers might consider analyzing impacts on early (pre-pandemic) versus late (post-pandemic) study participants. Subgroups based on age or health risk are also relevant because of the greater risk of becoming very ill among the elderly or those who are immune-compromised.

(2) Account for geography-based impact variation. It is possible that participants living in cities or states that are greatly impacted might have different intervention responses from those living in areas with a less restrictive COVID-19 response.

(3) Test for interaction effects. It is possible thatCOVID-19 directly interacts with the intervention being studied, especially if the intervention is being implemented in a healthcare setting.

(4) Use appropriate weights. Study teams may need to revise statistical analyses to weight or adjust for biases that may exist.

(5) Interpret with caution. When interpreting the data and findings, study teams should take into account this historical context, especially when data are pooled pre- and post-COVID 19.

Summary

These are difficult times. Now more than ever,policymakersandotherdecision-makersneedreliable data and rigorous analyses tohelp arrest the spread of COVID-19 and complete their important research studies. Until the pandemic subsides, it is imperative to maintain quality, rigor, and scientific integrity in research studies while following—and adjusting to—the ever-changing landscape of the COVID-19 pandemic.