One of the biggest promises of the “data age” is the creation of tools that will allow us to keep track and evaluate the performance of different projects and policies more efficiently. The main appeal of using Big data and AI for monitoring and evaluation (M&E) lies within the opportunity of having access to high-quality, timely, and accessible data to complement traditional sources (e.g. national statistics and surveys). Consequently, after the creation of the 2030 Sustainable Development Goals Agenda in 2015, the reviewing High-Level Panel made a call for a “data revolution”  in an effort to improve the quality of official statistics and open government information available to citizens. The goal was and is to empower the population—and stakeholders with vested interests in particular indicators—to monitor the progress towards specific targets.

Since then, the message of using Big data and AI to monitor the SDGs continues to be reverberated by organizations across the world. The advantage most commonly cited is the opportunity to shorten the feedback loop between policy formulation, monitoring, and implementation thanks to real-time data (where available) that can be leveraged by development actors to accelerate the speed and scale at which they implement changes. In furthering this movement, different agencies such as the UN Global Pulse, the European Commission, and MERL Tech have launched guides on how to apply Big data techniques to the field of development projects and policy evaluations. Although there has been an existing gap between the two fields—M&E and Big data analysis—due to challenges in the intersection between the two, evaluators such as Caroline Heder from the World Bank advocate for taking advantage of new technology to handle a lot more data, make sense of it and find new ways to visualize it.

Several projects are doing precisely that: using Big data for M&E strategies to tackle a myriad of issues. For example, Pulse Lab Jakarta has been working on integrating non-traditional real-time data sources to inform decision-makers following a crisis or a natural risk disaster; and Karsten Donnay, professor at the University of Zurich, carried out an investigation based on the use of Big data to monitor political instability in different countries. Through her study, she shows that leveraging this kind of resources can give policymakers and practitioners the ability to detect early warning signs as to where and when political instability may arise, which in turn would enable them to implement timely responses and interventions. 

It is important to highlight that in the context of the COVID-19 pandemic, the relevance of Big data and AI for monitoring and evaluation has increased due to the necessity of having real-time data on the disease outbreak and its impacts. Such data is crucial for governments to take the best possible courses of action and response policies. These approaches can be quite diverse, from exploring people’s reactions to the virus on social media, to generating data-driven models to evaluate COVID-19 interventions, tracking hospital bed capacity, and working on molecular modeling to evaluate what we know about the virus by comparing data with pre-existing information in pharmacological and genomic databases.

With the role of Big Data and AI being more relevant and critical than ever, this month’s Links We Like offers a list of key resources that shed light on how to best use these tools for monitoring and evaluation processes, specially applied to policy-making and development projects.