LWL #27: The Role of Big Data and AI in monitoring and evaluation (M&E)

Ivette Yáñez Soria, Mariana Rozo, Mariem Omrani, Sara Ortiz May 05 2021 Blog

LINKS WE LIKE #27

The Role of Big Data and AI in monitoring and evaluation (M&E)

One of the biggest promises of the “data age” is the creation of tools that will allow us to keep track and evaluate the performance of different projects and policies more efficiently. The main appeal of using Big data and AI for monitoring and evaluation (M&E) lies within the opportunity of having access to high-quality, timely, and accessible data to complement traditional sources (e.g. national statistics and surveys). Consequently, after the creation of the 2030 Sustainable Development Goals Agenda in 2015, the reviewing High-Level Panel made a call for a “data revolution”  in an effort to improve the quality of official statistics and open government information available to citizens. The goal was and is to empower the populationand stakeholders with vested interests in particular indicatorsto monitor the progress towards specific targets.

Since then, the message of using Big data and AI to monitor the SDGs continues to be reverberated by organizations across the world. The advantage most commonly cited is the opportunity to shorten the feedback loop between policy formulation, monitoring, and implementation thanks to real-time data (where available) that can be leveraged by development actors to accelerate the speed and scale at which they implement changes. In furthering this movement, different agencies such as the UN Global Pulse, the European Commission, and MERL Tech have launched guides on how to apply Big data techniques to the field of development projects and policy evaluations. Although there has been an existing gap between the two fieldsM&E and Big data analysisdue to challenges in the intersection between the two, evaluators such as Caroline Heder from the World Bank advocate for taking advantage of new technology to handle a lot more data, make sense of it and find new ways to visualize it.

Several projects are doing precisely that: using Big data for M&E strategies to tackle a myriad of issues. For example, Pulse Lab Jakarta has been working on integrating non-traditional real-time data sources to inform decision-makers following a crisis or a natural risk disaster; and Karsten Donnay, professor at the University of Zurich, carried out an investigation based on the use of Big data to monitor political instability in different countries. Through her study, she shows that leveraging this kind of resources can give policymakers and practitioners the ability to detect early warning signs as to where and when political instability may arise, which in turn would enable them to implement timely responses and interventions. 

It is important to highlight that in the context of the COVID-19 pandemic, the relevance of Big data and AI for monitoring and evaluation has increased due to the necessity of having real-time data on the disease outbreak and its impacts. Such data is crucial for governments to take the best possible courses of action and response policies. These approaches can be quite diverse, from exploring people’s reactions to the virus on social media, to generating data-driven models to evaluate COVID-19 interventions, tracking hospital bed capacity, and working on molecular modeling to evaluate what we know about the virus by comparing data with pre-existing information in pharmacological and genomic databases.

With the role of Big Data and AI being more relevant and critical than ever, this month’s Links We Like offers a list of key resources that shed light on how to best use these tools for monitoring and evaluation processes, specially applied to policy-making and development projects. 

To explore the ways in which Big data can contribute to best evaluate the SDG’s,  3ie (International Initiative for Impact Evaluation) carried out a webinar with Flowminder’s executive director Linus Bengtsson, 3ie senior evaluation specialists, Francis Rthainam, and lead in UN Jakarta Global Pulse, Sriganesh Lokanathan. The webinar’s panellists discussed different initiatives from their organizations where they have utilized Big data and AI models to measure and evaluate outcomes in challenging developing-country contexts. UN Jakarta Global Pulse, for instance, explored how remote sensing data coupled with agricultural surveys and census data could lead to a better understanding of whether the existing rice farming practices in Indonesia are contributing to a higher productivity. Flowminder, on the other hand, has focused on using satellite imagery to produce high resolution maps that show Nigerians school coverage or the population of Nepal, as well as on using cell phone data to track the mobility changes that arose during the confinement period for COVID-19 in Ghana. Through these examples the panellists emphasized how these new data techniques allowed them to have accurate, detailed and useful information that would have been otherwise very expensive and time-consuming (almost impossible) to obtain through traditional data methods. They reiterated that the potential of non-traditional data for M&E processes is tremendous, and while it does not replace traditional data, it can complement it to a great extent.

In his article, John Mataya from the USAID Learning Lab reflects on the three most impactful changes brought about by new efforts to monitor and evaluate development driven by data science. The first is in relation to surveys, where the author observes an incremental use of satellite images and mobile phone data alongside small-scale surveys to monitor and evaluate development projects. Particularly, this new modality can be very helpful in topics such as migrations, famine, conflict, and refugees, where the access to near real-time data, can guide and inform decision-makers’ actions and interventions. The second change has to do with the use of text analytics through Python and R to scrape and transform different texts into meaningful insights. Although this technique is not as ample, over time the resources available for this method of analysis will become more widely adopted. Lastly, the article mentions the use of computer vision, or facial detection as a way to monitor the response and participation at events, training, and conferences beyond the more traditional approaches that are usually limited to giving a total count of who participated in a given event. Through this tool, we could understand the audience’s reaction and perception to the information they are receiving, and discover new ways to engage people and even beneficiaries of international development projects.

Through a very detailed and thorough report commissioned by the Rockefeller Foundation, Michael Bamberger and Peter York unpack the ways in which data science can contribute to the field of evaluation and monitoring for development. The report argues program evaluation and data science have to join efforts to understand, in a more rapid and cost-effective fashion, whether a development project is working, what has improved,and how to scale the positive outcomes. They place emphasis on the opportunities this era brings forth for more accurate and timely M&E of development programs, but also on the challenges of merging two fields that have historically been separate. For example, the complexity of the programs can pose a difficulty in collecting data for all the relevant contextual factors. There is also the risk of failing to predict unintended outcomes when the focus is only or primarily placed on monitoring the intended outcomes of the project. Furthermore, there are issues related to data access and management such as collecting data for hard-to-reach groups, integrating different data sources, avoiding sample selection bias, and reducing the cost of data collection and analysis. To address these challenges, the report identifies a set of recommendations to facilitate the integration between data science and M&E. Among these suggestions are the creation of a conducive policy environment that has regulations for the access to public data; considerations for privacy and confidentiality and algorithmic transparency; the promotion of  available open data; establishing access to computing facilities and technical analysis expertise, and finally, creating organizational structures that promote the integration of the two fields.

The Vivacity project is an initiative that started five years ago by a group of Cambridge graduates who wanted to improve urban infrastructure. Through their artificial intelligence sensors and Smart Junctions signal control they gather detailed and anonymous data 24/7 on transport modes, traffic flow, and travel patterns. By implementing these anonymized sensors they can monitor the mobility patterns of different types of vehicles and pedestrians. Essentially they are building smart cities by generating real-time prediction data, traffic signal control, and guiding connected and autonomous vehicles. This with the objective of evaluating the current mobility spaces and to create data driven and data-optimized strategic decisions to help local authorities and governments improve their transport network and urban infrastructure. During lockdown measures, Vivacity also evaluated and monitored the efficacy of its social distancing response to the COVID-19 crisis. Using their anonymized camera sensors they could determine if two persons were maintaining the 2 meters distancing measures over time and give governments greater insight into the execution of this action.

The Policy-Insider.AI App (PI.AI App) is an example of how AI systems have been used not only in the evaluation and monitoring of public policy but also in empowering civil society to track policy worldwide. Created by AI experts, the PI.AI App centralizes global public policy and opens it up to stakeholder monitoring and analysis. This app offers users the possibility of enabling better and quicker access to policy. One of its goals is to promote transparency in policy-making and simplifying the process of searching relevant policy data. Nowadays, PI.AI is being used by NGOs, businesses, media outlets, academics, policy-makers and political parties. By leveraging AI, the app monitors policy institutions currently in Europe and some in the USA and facilitates  decision-making anywhere and anytime, while also translating to a preferred language and sending alerts and notifications. Overall, AI used in apps like this one  enhances transparency and policy monitoring.

In response to the pandemic, the Government of Togo launched “Novissi,” an innovative social assistance program described as an “exemplary case of social protection in response to the coronavirus pandemic in Africa.” Researchers from the University of California, Berkeley and Innovations for Poverty Action, GiveDirectly and the Government of Togo used satellite imagery and mobile phone data and metadata to accurately target the beneficiaries. First, they trained a machine-learning algorithm to estimate the wealth of very small regions (i.e., 2.4 km tiles) based on their geographic characteristics (e.g. places with metal roofs and high-quality roads are indicative of wealth). Once the geographic areas were mapped, researchers used mobile metadata to target the individuals with the greatest need. Afterwards, a second machine learning algorithm was trained to predict consumption from mobile phone metadata for each of the 5.7 million mobile subscribers in the country. The consumption estimates represent a proxy for poverty and therefore facilitate the targeting of beneficiaries. This poverty mapping and targeting exercise exemplifies how harnessing big data can improve projects and policy evaluation.

Further Afield

Share