Decision-based Fact-making: How it Really Works

It’s one thing to talk about how great it is to combine different kinds of data in your research, its altogether another matter of making it work in reality and then feeding all of it into a policy process that is not big on evidence based decision making. So what might a policy relevant project look like that is both qualitative and quantitative in scope? Well we’ve been answering that question for a while now in a policy relevant context. And these were some of the simplified assertions we came up with. Our context is measuring governance and democracy but this might apply to other areas of research as well like state failure, terrorism and so on.

The value of country-specific structural data

Structural data, such as GDP per capita, political indices and human rights measures, provide a sturdy platform on which to build country analysis. Structural data are compiled by recognised organizations, sometimes in partnership with host nations. Structural data allows the end user to rank countries for quick assessments of performance within sub-sectors. Country level structural data also enable comparative analysis. For example, one may compare the voting rates among women in Ghana and Cote D’Ivoire using data collected by the UNDP or the World Bank. Using the same indicators and econometric analysis it is possible to determine in what way women’s voting rates in Ghana and Cote D’Ivoire are influenced by education levels, rural and urban environments, and formal employment Many statistical indicators are themselves composite indices capturing several underlying concepts in a single score. The UNDP’s Human Development Index (HDI) is an example of a composite index. Indexing makes quantitative data easy to handle and compare and is useful for broad strategic evaluation across countries. For example, the CIFP indexing approach utilises a three-step process of initially collecting data on a yearly basis, assigning raw scores a global rank based upon a continuous distribution of countries for each indicator and then ranking countries for a specific year.

The limits of structural data

Structural data has obvious merit from a macro or strategic perspective but, a number of factors limit its utility as the sole source of information in measuring governance and democracy. At the sub-national level variations in both the types and method of data collection tend to limit an end-user’s ability to compare governance indicators across sub-regions or within a single region over time. In particular, sub-national data is often not delineated by age or gender, thus limiting the extent to which it can inform targeted development programming. Time lags are also an issue.

The value of dynamic data analysis

The systematic collection and evaluation of dynamic data also known as events-based information analysis, is highly relevant to policy programming. Dynamic data analysis whether it draws on information from media sources or country experts, is useful for identifying up-to-date trends in popular perceptions, preferences and stakeholder behaviours. Dynamic data analysis can add considerable value through regularized and standardised reporting. It can deepen understanding of trends found in structural data, and can highlight trend reversals. For example, a statistical study may show a steady decline in violent events over a series of years, but current events may evidence a sudden surge in violent demonstrations, one that will show up in structural data only until after the fact. Events-based information can also provide a window into stakeholder perceptions, how they are reacting to real-time changes and why they are doing so.

Events data draws from a myriad of open sources collected by humans or through machine-coded language. In either case, when each discrete event is analyzed in a structured and systematic fashion, patterns of performance begin to emerge. Pattern recognition is especially important to the analyst who is engaged in continuous country monitoring and, who wants to make projections about short-term changes within a country on the basis of recent trends. Consider patterns of decline in governance performance approximately a year prior to the declared state of emergency in Pakistan in 2007. Clearly, in the case of Pakistan, there was considerable evidence of an approaching crisis. Such evidence, if properly understood, can allow policymakers to respond in a timely fashion to impending problems, rather than simply responding after the fact.

The limits of dynamic data analysis

Events-based information cannot provide the entire context for complex situations, nor does it necessarily provide a complete representation of root causes. Events represent specific interactions among key players, known as stakeholders, in a given situation. Discrete events can act as accelerants (factors that tend to magnify the effects of existing issues) either on their own or in conjunction with a series of similar events. Events can also be triggers that provide the immediate proximate cause for crises. Events precipitate reactions and provided that the appropriate pre-conditions are in place become the basis for wholesale transformation. An extreme but illustrative example was the assassination of Juvenal Habyarimana, the President of Rwanda, was a trigger for the ensuing genocide in Rwanda, but without the underlying structural tensions deriving from ethnic fragmentation, power imbalances, and land shortages among other things, that radical elements could exploit, such a severe reaction most likely would not have occurred.

How dynamic data analysis can help the analyst

Given the frequent absence of opinion polls on matters relevant to governance and democracy, learning about popular reactions to events through media reports or other sources is an important way to understand the reality of governance and democracy on the ground and to ascertain how a government is responding to social, cultural, and economic pressures and opportunities. Seeking out different sources for a broader set of views can of course reduce the bias that might occur were one to rely on a single media source for all information. Local language media are also important sources of information. All reports of an event will most likely provide useful information and ideas, but the analyst must decide what is “fact”, what is “conjecture” and what is “polemic”, and how reliable event coverage is as a result.  Analysts need to look at events in the context of past history and social realities, to try to understand what is driving an event.

The usefulness of expert opinions, surveys and polling

Human insight can offer invaluably nuanced views. Qualitative information, of this kind is a valuable complement to the systematic collection of statistical data, as it uncovers details and nuance. Put simply, when correctly structured, expert opinion can provide the “why” behind the “what” revealed through structural and dynamic data analysis. Expert opinions can provide detailed insight into specific issue areas, as well as offer ideas about what areas deserve the most attention going forward, either because they are functioning well and can be used to propagate positive reform in other parts of the governance system, or because they are weakening and threaten to undermine stability and development in other sectors. For example, CIFP’s expert survey on Ghana highlights the problem of low popular expectations of government as an obstacle to improving governance performance. Ghanaians have become so accustomed to limited government capacity that they have ceased to seriously challenge the government on its service delivery. The experts consulted for this study suggested programming aimed at popular democratic education as a way to counter these issues. Both problem and solution would have been difficult if not impossible to discern in the absence of expert opinion.

However, once identified, structural data, in the form of educational performance statistics, popular surveys, and other quantitative indicators, can be enlisted to measure progress towards achievement of the articulated goal. As discussed above, statistical analyses are limited by the ways that they can be and are operationalized. Qualitative information minimises this problem by describing the whole of a situation in detail, including all the bits and pieces that are difficult to include in a statistical analysis. For example, a human rights expert with long experience in a country can provide a full picture of the local rights environment, bringing in elements of culture, history, and analogous situations. Expert opinions also provide a valuable challenge function to quantitative analysis. If enough experts tell a story that differs with a statistical snapshot it can be worthwhile reconsidering the validity of a quantitative-based conclusion, and potentially revising the selection and/or operationalization of quantitative indicators.

The limits to expert opinions, surveys and polling

Individual expert opinion tends to reveal only one part of a larger picture. People have subjective viewpoints, whether they are ordinary citizens or specialists. Specialists are likely to overestimate the importance of their field to the overall governance and democratic situation in a country. Ideology can cloud opinions, as can personal experience and bias. Expert opinions cannot provide an objectively true description of a country’s governance and democratic processes. Expert opinions have other limitations as well. Research processes involving expert opinions are often comparatively expensive relative to other methods; many experts often require compensation, and it can be costly to interview a sufficient number to validate the information collected. In addition, excessive consultation of experts can actually burden the country that analysts are seeking to help. Research takes the time of the experts themselves, many of whom are involved in vital positions within their countries’ government and society; when confronted with endless requests for interviews by international researchers, their own work may suffer.

Using expert opinions, surveys and polling effectively

It is crucial to derive expert-based information from as wide a base as possible to take advantage of multiple viewpoints and to limit the potential for ‘tunnel-vision’ and group-think that can arise from discussing ideas within a limited community that approaches problems from the same perspective. Subjective bias can never be eliminated altogether, but talking to a broad sample of people provides an analyst with a better overall conceptualisation of the society and its relationship with its government.  Expert panels should include individuals that approach different elements of governance, and approach them from different ideological and professional perspectives.

Combining information streams for better analysis

Meaningful analysis will include as much information as possible, ideally from all three of the sources described above. There are three main benefits to systematically combining information streams:

Challenge function: If different streams tell different stories one can investigate further to discover which is correct; analysis based upon a single information stream is impossible to verify or validate.

Depth of coverage: No single stream can provide a complete picture of a country’s democratic and governance performance. Combining streams can give more information at different levels.

Finer focus: Looking at different streams lets the analyst see more clearly what factors contribute most to outcomes. This idea connects to the previous two advantages of multi-stream analysis. If all three streams suggest that a certain factor is the key driver of a certain phenomenon one can have confidence that this is the case.

However, if there is disagreement one can investigate further to gain clarity and find out which angle is less accurate. Likewise, deeper coverage will allow a more detailed picture of how various factors interact to produce outcomes, allowing causal relationships to go beyond simple cause and effect to embrace network dynamics.

David Carment

Extracted from the

Measuring Governance and Democracy Processes Handbook ( With acknowledgements to Stewart Prest and Keven Wyjad – two graduates of the programme and researchers in the CIFP governance and democracy project.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s