Editor's Note: This blog was first published on 3ie's blog, Evidence Matters.
Geospatial impact evaluations offer deeper insights but also come with their unique set of challenges. As part of the GeoField community of practice, we are finding ways to unpack and address these by collaborating with individuals and organizations working across sectors. To that end, 3ie, AidData, and DevGlobal recently co-hosted a workshop with GIS experts to address key issues and best practices in response to technical questions received from participants before and during the session.
Increased availability of and access to high-frequency, granular geospatial data has allowed us to expand on insights from traditional evaluations and conduct them more frequently, efficiently, and cost-effectively. This has been particularly crucial in cases where data previously did not exist or is difficult to obtain through traditional data collection methods. At the same time, geospatial data can be misinterpreted or misrepresented in many ways. For example, NDVI, a common proxy indicator for vegetation cover and crop yield, is not always the best measure for these outcomes, and the most relevant proxy indicator may change based on the context (e.g., SAVI is more sensitive to vegetation cover in dry regions).
Similarly, feeding training data from one region into a Machine Learning (ML) model to classify land use in an entirely different region is likely to be inaccurate and misleading. While some aspects of these technical issues are relatively clear-cut to address, many are less obvious. For instance, we can check how well our ML models perform in different contexts, particularly when incorporating context-specific on-the-ground data for validation, but what level of accuracy is “good enough”?
Another aspect that makes geospatial evaluations particularly challenging is that they are by nature interdisciplinary and require a strong technical understanding of not only econometrics and statistics, but also geospatial analysis and the specific sector and context. Therefore, while we may know how to deal with an issue within one discipline, it is likely approached differently in another, and the overlap is not immediately clear. For example, spatial autocorrelation in econometrics is typically addressed by clustering standard errors. However, with extremely granular environmental data without clear administrative or social boundaries on which to cluster data, how should we address any existing autocorrelation?
During the recent Geospatial Technical Workshop, experts from various fields convened virtually to discuss and provide one another with feedback on practical technical issues faced in implementing their geospatial impact evaluations in multiple sectors.
Technical questions were submitted and shared before the event, and participants had the opportunity to jump into the conversation to suggest ways forward during Q&A and breakout sessions. The topics covered included best practices, approaches, and ideas for:
The shared commitment and foresight to engage with the questions demonstrate that shying away from these issues is not an option. While there were no clear-cut answers to many questions, the session takeaways (read the discussion summary here) included some new techniques to try, a better understanding of the gaps in our collective knowledge, and better connections with the individuals and organizations that are eager to push the technical frontier—and learn and advance together.
Asking tough questions and getting answers in this format is not easy, but the value of bringing a wide range of experts and policymakers into one room to understand and tackle them is immense. Keep an eye out for future events and workshops where we can reconnect, or feel free to contact info@3ieimpact.org to discuss or propose your ideas for an event.
Fiona Kastel is a Senior Research Associate at 3ie. Fiona provides research, data analytics, training, and program management support for multiple programs and initiatives on agriculture, education, finance, health, and policy and institutional reform.
Sanchi Lokhande is a Research Associate at 3ie, providing research and business development support.
Anca Dumitrescu is a Lead Evaluation Specialist at 3ie. She manages the West Africa Capacity-building and Impact Evaluation (WACIE) program. She also supports membership activities with 3ie’s francophone members, contributes to business development initiatives and provides technical support for other Washington Office programs.
Devika Lakhote is a Data Scientist at 3ie. She provides technical and management leadership and support in the application of data science to impact evaluations and evidence synthesis.
Douglas Glandon is Chief of Strategic Initiatives at 3ie, including in data innovations, cost evidence, and training. Douglas guides and implements 3ie's strategic direction, with an emphasis on technical innovation and efficient, high-quality delivery of services to 3ie's partners.
