Can Developmental Evaluation Survive Going Remote?
May 12, 2021 | By By Chris Thompson, Social Impact Chief of Party for the USAID Jalin DE
One of the COVID-19 pandemic’s many legacies has been an accelerated shift to remote work as societies lockdown and socially distance. While working remotely affects many of us in the evaluation sciences, it poses a unique challenge to one of the discipline’s newest and increasingly popular approaches, developmental evaluation (DE).
USAID designed the Jalin Project (“Jalin” means “intertwined” in Bahasa Indonesia) to support the Ministry of Health (MOH) to accelerate reduction in preventable maternal and newborn deaths by working with local governments, the private sector, and civil society to co-define, co-create, and co-finance MNH innovations in the six provinces of Banten, Central Java, East Java, North Sumatera, South Sulawesi, and West Java.
DE seeks to strengthen programs by enhancing innovation and adaptation, in part through physically embedding evaluators within projects. While evaluators remain independent, they also integrate to access a team’s processes, make observations, and guide reflection.
Most DE manuals, including Dr. Michael Quinn Patton’s landmark 2011 Developmental Evaluation, emphasize face-to-face approaches for embedding successfully, like being present at meetings, having water cooler conversations, and—my favorite—getting coffee with stakeholders, decision makers, and well, basically everyone. The need to socially distance and work remotely to limit COVID-19’s threat to public health has limited the utility of many of these tactics and, unfortunately, my caffeine intake.
Mentoring activity in Puskesmas | Photo Credit: Oscar Siagian, USAID Jalin
So, when USAID/Indonesia’s DE for USAID Jalin – a three-year DE to support innovation in maternal and newborn health (MNH) in Indonesia – began teleworking due to the pandemic, my team and I reached out to other DEs which had gone remote. We sought to learn what happens when an approach that prides itself on in-person connections begins to work virtually. Here are a few “takeways.”
Takeaway #1: Everyone’s Doing It
The first thing we learned was that many developmental evaluators globally had started working remotely and faced similar challenges. These included Social Impact’s DEs of USAID’s Global Digital Strategy implementation as part of the USAID Developmental Evaluation Pilot Activity (DEPA-MERL) and UNICEF’s COVID-19 global response. Others which had gone remote include the USAID/CIRCLE project’s DE of the Boresha Afya program in Tanzania, the Gates Foundation’s DE of the Jilinde Program in Kenya, and an DE with UNFPA.
Realizing we were not facing this challenge alone, the DE for USAID Jalin distilled its conversations with these DEs to create a Guide to Remote Developmental Evaluation.
Takeaway #2: Benefits Exist, Despite the Challenges
The Guide describes tangible benefits to evaluating remotely that may not have been initially apparent. One advantage is that shifting to remote work reduces costs. While DE budgets vary depending on the program and research questions, DEPA-MERL found that pilots range from $300K-$1.7M because evaluators embed with teams for extended periods. The Guide found that evaluators were reinvesting savings on travel and other direct costs into communications and non-traditional, impactful deliverables like animated videos and online maps.
Crucially, for an approach with a need for real time data, DEs were finding it easier to reach and collaborate with more people (particularly around diverse geographic settings) after going remote. As one evaluator with CIRCLE put it, “We used to meet just a few people because of competing physical engagements. But virtually, we get almost everybody participating.” Even the DE for USAID Jalin found that health workers – despite being involved in COVID-19 emergency response – were just as willing now to discuss MNH by phone.
The Guide also shows that the interconnected relationship between embeddedness, trust, engagement, and data collection remains undiminished for remote DEs. Success at embedding is a function of and in turn strengthens trust. Becoming a trusted team member improves an evaluator’s ability to collaborate and collect information to inform decision-making. Though evaluators struggled in these areas in a remote environment, they also found ways to leverage remote work to their advantage by maximizing virtual workspaces, connecting more frequently with donors and stakeholders, and extending timeframes for data collection.
Takeaway #3: Interest is High
An active audience exists for learning how to successfully implement remote DEs. Dr. Quinn Patton hosted a webinar through his Blue Marble Evaluation network on February 25th to launch the Guide which engaged 220 participants from five continents (view the recording here). The audience was split between evaluators and donors, including USAID, ASEAN, UN, the Bill & Melinda Gates Foundation, and Clinton Health Access Initiative. A post-event survey indicated that 90 percent of these participants learned something new!
However, it is important to distinguish between thriving and surviving. As professional evaluators, we are motivated by advancing development effectiveness in the face of challenges. But, is DE really better off done remote rather than in-person? This question is one for serious discussion as the pandemic subsides and a return to offices becomes possible, but for now, one thing is undeniable…
Practitioners and funders continue to find value in DE despite the challenges caused by COVID-19, and this bodes well for this relatively new branch of evaluation sciences, whether done remote or returning to in-person just as some of us are doing in our real lives. Speaking of which, I’m off to the coffee shop. I never did figure out my pandemic-purchased, home espresso maker.
For more information and detail, please see the Guide to Remote Developmental Evaluation that the DE for USAID Jalin prepared to assist funders and practitioners to successfully implement remote DEs.
Photo Credit: Oscar Siagian, USAID Jalin
Graphic Credit: Reza Istafi