Rick On the Road

Evaluation quality standards: Theories in need of testing?

2011-10-24T10:46:00.002Z

Since the beginning of this year I havebeen part of a DFID funded exercise which has the aim of “Developing a broaderrange of rigorous designs and methods for impact evaluations” Part of the briefhas been to develop draft quality standards, to help identify “the differencebetween appropriate, high quality use of the approach and inappropriate/ poorquality use”

A quick search of what already exists suggeststhat there is no shortage of quality standards. Those relevant todevelopment projects have been listed onlinehere. They include:

Standards agreed by multiple organisations,e.g. OECD-DAC and various national evaluation societies. The former are ofinterest to aid organisations where as the latter are of more interest toevaluators.
Standards developed for usewithin individual organisations, e.g. DFID and EuropeAID
Methodology specific standards,e.g. those relating to randomised and other kinds of experimental methods, andqualitative research

In addition there is a much larger body of academicliterature on the use and mis-use of various more specific methods.

A scan of the criteria I have listed shows thata variety of types of evaluation criteria are used, including:

Process criteria, where thefocus is on how evaluations are done. e.g. relevance, timeliness,accessibility, inclusiveness
Normative criteria, where thefocus is on principles of behaviour e.g. independence, impartiality, ethicality
Technical criteria, where thefocus is on attributes of the methods used e.g. reliability and validity

Somewhat surprisingly, technical criterialike reliability and validity are in the minority, being two of at least 20OECD-DAC criteria. The more encompassing topic of Evaluation Design is only one ofthe 17 main topics in the DFID Quality Assurance template for revising draftevaluations. There are three possible reasons why this is so: (a) Processattributes may be more important, in terms of their effects on what happens toan evaluation, during and after its production, (b) It is hard to identifygeneric quality criteria for a diversity of evaluation methodologies, (c) Lists have nosize limits. For example, the DFID QA template has 85 subsidiary questions under 17 main topics.

Given these circumstances what is the bestway forward, of addressing the need for quality standards for “a broader rangeof rigorous designs and methods for impact evaluations”? The first step mightbe to develop specific guidance which can be packed in separate notes onparticular evaluation designs and methods. The primary problem may be simplelack of knowledge about the methods available; knowing how to choose betweenthem may be in fact “a problem we would like to have”, which needs to beaddressed after people at least know something about the alternative methods.The Asian Development Bank has addressed this issue through its “Knowledge Solutions”series of publications.

The second step that could be taken wouldbe to develop more generic guidance that can be incorporated into the existingquality standards. Our initial proposal focused on developing some additionaldesign focused quality standards that could be used with some reliabilityacross different users. But perhaps this is a side issue. Finding out whatquality criteria really matter, may be more important. However, there seems tobe very little evidence on what quality attributes matter. In 2008 Forss et alcarried out a study: “AreSida Evaluations Good Enough? An Assessment of 34 Evaluation Reports” The authorsgathered and analysed empirical data on 40 different quality attributes of evaluationreports published between 2003 and 2005. Despite suggestions made, the reportwas not required to examine the relationship between these attributes and thesubsequent use of the evaluations. Yet, the insufficient use of evaluations hasbeen a long standing concern to evaluators and to those funding evaluations.

There are at least 4 different hypotheses that would be worth testing in futureversions of the SIDA study that did look at evaluation quality and usage:

Quality is largely irrelevant, what matters is how the evaluationresults are communicated.
Quality matters, especially the use of a rigorous methodology, whichis able to address attribution issues
Quality matters, especially the use of participatory processes thatengage stakeholders
Quality matters, but it is a multi-dimensional issue. The more dimensionsare addressed, the more likely that the evaluation results will be used.

The first is in effect the null hypothesis,and one which needs to be taken seriously. The second hypothesis seems to bethe position taken by 3ie and other advocates of RCTs and their next-best substitutes.It could be described as the killer assumption being made by RCT advocates that is yet to be tested. Thethird could be the position of some of the supporters of the “BigPush Back” against inappropriate demands for performance reporting. Thefourth is the view present in the OECD-DAC evaluation standards, which can beread as a narrative theory of change about how a complex of evaluation qualityfeatures will lead to evaluation use, strengthened accountability, contributeto learning and improved development outcomes. I have taken the liberty of identifyingthe various possible causal connections in that theory of change in this network diagram below. As notedabove, one interesting feature is that the attributes of reliability andvalidity are only one part of a much bigger picture.

[Click on image to view alarger version of the diagram]

Whilewe wait for the evidence…

We should considertransparency as a pre-eminent quality criterion, which would be applicableacross all types of evaluation designs. It is a meta-quality, enablingjudgments about other qualities. It also addresses the issue of robustness,which was of concern to DFID. The more explicit and articulated an evaluationdesign is, the more vulnerable it will be to criticism and identification oferror. Robust designs will be those that can survive this process. This view connects to wider ideas in thephilosophy of science about the importance of falsifiablity as aquality of scientific theories (promoted by Popper and others).

Transparency might be expected at both amacro and micro level. At the macro level, we might ask these types of qualityassurance questions:

Before the evaluation: Has an evaluation plan been lodged, whichincludes the hypotheses to be tested? Doing so will help reduce selectivereporting and opportunistic data mining
After the evaluation: Is the evaluation report available? Is the rawdata available for re-analysis using the same or different methods?

Substantial progressis now being made with the availability of evaluation reports. Some bilateralagencies are considering the use of evaluation/trial registries, which are increasinglycommonplace in some field of research. However, availability of raw data seemslikely to remain the most challenging requirement for many evaluators.

At the micro-level,more transparency could be expected in the particular contents of evaluationplans and reports. The DFID Quality Assurance templates seem to be mostoperationalised set of evaluation quality standards available at present. Thefollowing types of questions could be considered for inclusion in thosetemplates:

Is itclear how specific features of the project/program influenced the evaluationdesign?
Haverejected evaluation design choices been explained?
Have termslike impact been clearly defined?
What kindsof impact were examined?
Whereattribution is claimed is there also a plausible explanations of the causalprocesses at work?
Have distinctions been made between causes which are necessary, sufficient orneither (but still contributory)?
Are thereassessments of what would have happened without the intervention?

This approach seems tohave some support in other spheres of evaluation work, not associated withdevelopment aid: “The transparency, orclarity, in the reporting of individual studies is key” TREND statement, 2004

In summary, three mainrecommendations have been made above:

Developtechnical guidance notes, separate from additional quality criteria
Identifyspecific areas where transparency of evaluation designs and methods isessential, for possible inclusion in DFID QA templates, and the like
Seek anduse opportunities to test out the relevance of different evaluation criteria,in terms of their effects on evaluationuse

PS: This text was thebasis of one of the presentations to DFID staff (and others) in a workshop on 7^thOctober 2011 on the subject of “Developing a broaderrange of rigorous designs and methods for impact evaluations” The views expressed above are my own and should not be taken to reflect the views of either DFID or others involved in the exercise.

Relative rather than absolute counterfactuals: A more useful alternative?

2011-09-04T16:41:00.000Z

Background

The basic design of a randomised control trial (RCT) involves comparisons of two groups: an intervention (or “treatment”) group and a control group, at two points of time, before an intervention begins and after the intervention ends. The expectation (hypothesis) is that there will be a bigger change on an agreed impact measure in the intervention group than in the control group. This hypothesis can be tested by comparing the average change in the impact status of members of the two groups, and applying a statistical test to establish that this difference was unlikely to be a chance finding (e.g. less than 5% probability of being a chance difference). The two groups are made comparable by randomly assigning participants to both groups. The types of comparisons involved are shown in this fictional example below.

	A. Intervention group	B. Control group
Before intervention	Average income per household = $1000 year. N = 500	Average income per household = $1000 year N=500
After intervention	Average income per household = $1500 year. N = 500	Average income per household = $1200 year N=500
PS: See Comment 3 below re this table]	Difference over time = $500	Difference over time = $200
PS: See Comment 3 below re this table]	Difference between changes in A and B = £300

This method allows a comparison with what could be called an absolute counterfactual: what would have happened if there was no intervention.

Note that only the impact indicator is measured, there is no measurement of the intervention. This is because the intervention is assumed to be the same across all participants in the intervention group. This assumption is reasonable with some development interventions, such as those involving financial or medical activities (e.g. cash transfers or de-worming). Some information based interventions, using radio programs or the distribution of booklets, can also be assumed to be available to all participants in a standardised form. Where delivery is standardised it makes sense to measure the average impacts on the intervention and control group, because significant variations in impact are not expected to arise from the intervention.

Alternate views

There are however many development interventions where delivery is not expected to be standardised and where the opposite is the case, that delivery is expected to be customised. Here the agent delivering the intervention is expected to have some autonomy and to use that autonomy to the benefit of the participants. Examples of such agents would include community development workers, agricultural extension workers, teachers, nurses, midwives, nurses, doctors, plus all their supervisors. On a more collective level would be providers of training to such groups working in different locations. Also included would be almost all forms of technical assistance provided by development agencies.

In these settings measurement of the intervention, as well as the actual impact, will be essential before any conclusions can be drawn about attribution – the extent to which the intervention caused the observed impacts. Let us temporarily assume that it will be possible to come up with a measurement of the degree to which an intervention has been successfully implemented, a quality measure of some kind. It might be very crude, such as number of days an extension worker has spent in villages they are responsible for, or it might be a more sophisticated index combining multiple attributes of quality (e.g. weighted checklists).

Data on implementation quality and observed impact (i.e. an After minus a Before measure) can now be brought together in a two dimensional scatter plot. In this exercise there is no longer a control group, just an intervention group where implementation has been variable but measured. This provides an opportunity to explore the relative counterfactual, what would have happened if implementation was less successful, and less successful still, etc. In this situation we could hypothesise that if the intervention did cause the observed impacts then there would be a statistically significant correlation between the quality of implementation and observed impact. In place of an absolute counterfactual obtained via the use of control group, where there was no intervention we have relative counterfactuals, in the form of participants exposed to interventions of different qualities. In place of an average, we have a correlation.

There are a number of advantages to this approach. Firstly, with the same amount of evaluation funds available, the number of intervention cases that can be measured can be doubled, because a control group is no longer being used. In addition to obtaining (or not) a statistically significant correlation, we can also identify the strength of the relationship between the intervention and the impact. This will be visible in the slope of the regression line. A steep slope[1] would imply that small improvements in implementation can make big improvements in observed impacts and vice versa. If a non-lineal relationship is found then the shape of a best fitting regression line might also be informative, about where improvements will generate more versus less improvement.

Another typical feature of scatter plots is outliers. There may be some participants (individuals or groups of) who have received a high quality intervention, but where the impact has been modest, i.e. a negative outlier. Conversely, there may be some participants who have received a poor quality intervention, but where the impact has been impressive, i.e. a positive outlier. These are both important learning opportunities, which could be explored via the use of in-depth cases studies . But ideally these case studies would be informed by some theory, directing us where to look.

Evaluators sometimes talk about implementation failure versus theory failure. In her Genuine Evaluation blogPatricia Rogers gives an interesting example from Ghana, involving the distribution of Vitamin A tablets to women in order to reduce pregnancy related mortality rates. Contrary to previous findings, there was no significant impact. But as Patricia noted, the researchers appeared to have failed to measure compliance i.e. whether all the women actually took the tables given to them! This appears to be a serious case of implementation failure, in that the implementers could have designed a delivery mechanism that ensured compliance. Theory failure would be where our understanding of how Vitamin A affects women’s health appears to be faulty, because expected impacts do not materialise, after women have taken the prescribed medication.

In the argument developed so far, we have already proposed measuring quality of implementation, rather than making any assumptions about how it is happening. However, it is still possible that we might face “implementation measurement failure”. In other words, there may be some aspect of the implementation process that was not captured by the measure used, and which was causally connected to the conspicuous impact, or lack thereof. A case study, looking at the implementation process in the outlier cases might help us identify the missing dimension. Re-measurement of implementation success incorporating this dimension might produce a higher correlation result. If it did not, then we might by default then have a good reason to believe we are now dealing with theory failure, i.e. a lack of understanding of how an intervention has its impact. Again, case studies of the outliers could help generate hypotheses about these. Testing these out is likely to be more expensive than testing alternate views on implementation processes because data will be less readily at hand. For reasons of economy and practicality implementation failure should be our first suspect.

In addition to having informative outliers to explore, the use of a scatter plot enables us to identify another potential outcome not readily available via the use of control groups, where the focus is on averages. In some programmes poor implementation may not simply lead to no impact (i.e. no difference between the average impact of control and intervention groups). Poor implementation may lead to negative impacts. For example, a poorly managed savings and credit programme may lead to increased indebtedness in some communities. In a standard comparison between intervention and control groups this type of failure would usually need to be present in a large of cases before it became visible in a net negative average impact. In a scatter plot any negative cases would be immediately visible, including their relationship to implementation quality.

To summarise so far, the assumption about standardised delivery of an intervention does not fit the reality of many development programmes. Replacing assumptions by measurement will provide a much richer picture of the relationship between an intervention and the expected impacts. Overall impact can still be measured, by using a correlation coefficient. In addition we can see the potential for greater impact present in existing implementation practice (the slope of the regression line). We can also find outliers that can help improve our understanding of implementation and impact process. We can also quickly identify negative impacts, as well as the absence of any impact.

Perhaps more important still, the exploration of internal differences in implementation means that the autonomy of development agents can be valued and encouraged. Local experimentation might then generate more useful outliers, and not be seen simply as statistical noise. This is experimentation with a small e, of the kind advocated by Chris Blattman in his presentationto DFID on 1^st September 2011, and of a kind long advocated by most competent NGOs.

Given this discussion is about counterfactuals, it might be worth considering what would happen if this implementation measurement based approach was not used, where an intervention is being delivered in a non-standard way. One example is a quasi-experimental evaluation of an agricultural project in Tanzania, described in Oxfam GB‘s paper on its Global Performance Framework[2] . “Oxfam is working with local partners in four districts of Shinyanga Region, Tanzania, to support over 4,000 smallholder farmers (54% of whom are women) to enhance their production and marketing of local chicken and rice. To promote group cohesion and solidarity, the producers are encouraged to form themselves into savings and internal lending communities. They are also provided with specialised training and marketing supporting, including forming linkages with buyers through the establishment of collection centres.” This is a classic case where the staff of the partner organisations would need to exercise considerable judgement about how to best help each community. It is unlikely that each community was given a standard package of assistance, without any deliberate customisations nor any unintentional quality variations along the way. Nevertheless, the evaluation chose to measure the impact of the partner’s activities on changes in household incomes and women’s decision making power, by comparing the intervention group with a control group. Results of the two groups were described in terms of “% of targeted households living on more than £1.00 per day per capita”, and % of supported women are meaningfully involved in household decision making”. In using these measures to make comparisons Oxfam GB has effectively treated quality differences in the extension work as noise to be ignored, rather than as valuable information to be analysed. In the process they have unintentionally devalued the work of their partners.

A similar problem can be found elsewhere in the same document where Oxfam GB describes their new set of global outcome indicators. The Livelihood Support indicator is: % of targeted households living on more than £1.00 per day per capita (as used in the Tanzania example). In four of the six global indicators the unit of analysis are people, the ultimate intended beneficiaries of Oxfam GB’s work. However, the problem is that in most cases Oxfam GB does not work directly with such people. Instead Oxfam GB typically works with local NGOs who in turn work with such groups. In claiming to have increased the % of targeted households living on more than £1.00 per day per capita Oxfam GB is again obscuring through simplification the fact that it is those partners who are responsible for these achievements. Instead, I would argue that the unit of analysis many of Oxfam GB’s global outcome indicators should be the behaviour and performance of its partners. Its global indicator for Livelihood Support should read something like this: “x % of Oxfam GB partners working on rural livelihoods have managed to double the proportion of targeted households living on more than £1.00 per day per capita” Credit should be given to where credit is due. However, these kinds of claims will only be possible if and where Oxfam GB encourages partners to measure their implementation performance as well as changes taking place in the communities they are working with, and then to analyse the relationship between both measures.

Ironically, the suggestion to measure implementation sounds rather unfashionable and regressive, because we are often reading how in the past aid organisations used to focus too much on outputs and that now they need to focus more on impacts. But in practice it is not an either/or question. What we need is both, and both done well. Not something quickly produced by the Department of Rough Measures.

PS 4th September 2011: I forgot to discuss the issue of whether any form of randomisation would be useful where relative counterfactuals are being explored. In an absolute counterfactual experiment the recipients’ membership of control versus intervention groups is randomised. In a relative counterfactual “experiment” all participants will receive an intervention so there is no need to randomly assign participants to control versus intervention groups. But randomisation could be used to decide which staff worked with which participants (/vice versa). For example, where a single extension worker is assigned to a given community. But this would be less easily where a whole group of staff e.g. in a local health centre or local school, are responsible for the surrounding community.

Even where randomisation of staff was possible this would not prevent the impact of external factors influencing the impact of the intervention. It could be argued that the groups experiencing least impact and the poorest quality implementation were doing so, because of the influence of an independent cause (e.g. geographical isolation) that is not present amongst the groups experiencing bigger impacts and better quality implementation. Geographical isolation is a common exterbal influence in many rural development projects, one which is likely to make implementation of a livelihood initiative more difficult as well as making it more difficult for the participants to realise any benefits e.g. through sales of new produce at a regional market. Other external influences may affect the impact but not the intervention e.g. subsequent changes in market prices for produce. However, identifying the significance of external influences should be relatively easy, by making statistical tests of the difference in their prevalence in the high and low impact groups. This does of course require being able to identify potential external influences whereas as with randomised control trials (RCTs) no knowledge of other possible causes is needed (their influence is assumed to be equally distributed between control and intervention groups). However, this requirement could be considered as a "feature" rather than a "bug", because exploration of the role of other causal factors could inform and help improve implementation. On the other hand, the randomisation of control and intervention groups could encourage management's neglect of the role of other causal factors. There are clearly trade-offs here between competing evaluation quality criteria of rigour and utility.

[1]i.e. with observed impact on the y axis and intervention quality on the x axis

[2]Can we obtain the required rigour without randomisation? Oxfam GB’s non-experimental
Global Performance Framework. Karl Hughes, Claire Hutchings. August 2011

Evaluation methods looking for projects or projects seeking appropriate evaluation methods?

2011-08-16T13:19:00.016Z

A few months ago I carried out a brief desk review of 3ie's approach to funding impact evaluations, for AusAID's Office of Development Effectiveness. One question that I did not address was "Broadly, are there other organisations providing complementary approaches to 3ie for promoting quality evaluation to fill the evidence gap in international development?"

While my report for 3ie examined the pros and cons of 3ie's use of experimental methods as a preferred evaluation design, it did not look at the question of appropriate institutional structures for supporting better evaluations. Yet, you could argue that choices made about institutional structures could have more consequences than those involving the specifics of particular evaluation methods. The question quoted above seems to contain a tacit assumption about institutional arrangements, i.e that improvements in evaluation can best be promoted by funding externally located specialists centres of expertise, like 3ie. This kind of assumption seems questionable, for two sets of reasons that I explain below. One is to do with the results they generate, the other concerns the neglected potential of an alternative.

In the "3ie" (Open Window) model anyone can submit a proposal for an evaluation of a project implemented by any organisation. This approach is conducive to 'cherry picking' of evaluable (by experimental methods) projects and the collection of evaluations representing a miscellany of types of projects - about which it will be hard to generate useful generalisations. Plus an unknown number of other projects possibly being left unevaluated, because they dont fit the prevailing method preferences.

In the alternative scenario the funding of evaluations would not be outsourced to any specialist centre(s). Instead, an agency like DFID would identify a portfolio of projects needing evaluation. For example, those initiatives focusing on climate change adaptation. DFID would call for proposals for their evaluation and then screen those proposals, largely as it does now, but perhaps seeking a wider range of bidders.

Unlike the present process, it would then offer funding to the bidders who had provided, say the best 50% of, the proposals to develop those proposals further in more detail. At present there is no financial incentive to do so, and any time and money already spent on developing proposals is unlikely to be recompensed, because only one bidder will get the contract.

The expected result of this "proposal development" funding would be revised and expanded proposals that outlined the bidder's proposed methodology in considerable detail, in something like an inception report. All the bidders involved at this stage would need access to a given set of project documents and at least one collective meeting with the project holders.

The revised proposals would then be assessed by DFID, but with a much greater weighting towards the technical content of the proposal than exists at present. These second level assessment would benefit from the involvement of external specialists, as in the 3ie model. DFID Evaluation Department already does this in the case of some evaluations through the use of a quality assurance panel.The best proposal would then be funded as normal, and the evaluation then carried out.

Both the winning and losing technical proposals would then be put in the public domain via the DFID website in order to encourage cross fertilisation of ideas, external critiquing and public accountability. This is not the case at present. All bidders operate in isolation. There are no opportunities to learn from each other. The same appears to be the case with 3ie, the full text of technical proposals are not publicly available (even of those who were successful). Making the proposals public would mean that the proposal development funding had not been wasted, even where the proposals were not successful.

In summary, with the "external centre of expertise" model there is a risk that methodological preferences are the driving force behind what gets evaluated. The alternative is a portfolio-of-projects led approach, where interim funding support is used to generate a diversity of improved evaluation proposals, which are later made accessible by all and which can then inform future proposals.

A meta-evaluation might be useful to test the efficacy of this project-led approach. Other matched kinds of projects also needing evaluation could continue to be funded by the pre-existing mechanisms (e.g. in-country DFID offices). Pair comparisons could later be made of the quality of the evaluations that were subsequently produced by the two different mechanisms.Although it is likely there would be multiple points of difference, it should be possible for DFID, and any other stakeholders, to prioritise their relative importance, and come to an overall judgement of which has been most useful.

PS: 3ie seems to be heading in this direction, to some extent. 3ie now have a Policy Window where they have recently sought applications for the evaluation of projects belonging to a specific portfolio ("Poverty Reduction Interventions in Fiji" implemented by the Government of Fiji). Funding is available to cover costs of the successful bidder (only) to visit Fiji "to develop a scope of work to be included in a subsequent Request for Proposal (RFP) to conduct the impact evaluation".Subject to 3ie's approval of the developed proposal 3ie will then fund the implementation of the evaluation by the bidder.The success of this approach will be worth watching, especially its ability to ensure the evaluation of the whole portfolio of projects (which is likely to depend on 3ie having some flexiblity about the methodologies used). However, I am perhaps making a risky assumption here, that the projects within the portfolio to be evaluated have not already been pre-selected on the grounds of their suitability to 3ie's preferred approach.

PS: I have been reading the [Malawi] CIVIL SOCIETY GOVERNANCE FUND -
TECHNICAL SPECIFICATION REFERENCE DOCUMENT FOR POTENTIAL SERVICE PROVIDERS. In the section on the role of the Independent Evaluation Agent, it is stated that the agent will be responsible for "The commissioning and coordination of randomised control trials for two large projects funded during the first or second year of granting." This specification appears to have been made prior to the funding of any projects. So, will the fund managers feel obliged to find and fund two large projects that will be evaluable by RCTs? Fascinating, in a bizarre kind of way.

Models and reality: Dialogue through simulation

2011-04-08T14:09:00.036Z

I have been finalising preparations for a training workshop on network visualisation as an evaluation tool. In the process I came across this "Causality Map for the Enhanced Evaluation Framework", for Budget Support activities.

On the surface this diagram seems realistic, budget support is a complex process. However, I will probably use this diagram to highlight what is often missing in models of development interventions. Like many others, it lacks any feedback loops, and as such it is a model that is a long way from the reality it is trying to represent in summary form. Using a distinction being used more widely these days (and despite my reservations about it), I think this model qualifies as complicated but not complex. If you were to assign a numerical value to each node and to each connecting relationship, the value that would be generated at the end of the process (on the right) would always be the same.

The picture changes radically as soon as you include feedback loops, which is much easier to do when you use network rather than chain models (and where you give up using one dimension in the above type of diagram to represent the passage of time). Here below is my very simple example. This model represents five actors. They all start with a self-esteem rating of 1, but their subsequent self-esteem depends on the influence of the others they are connected to (represented by positive or negative link values, [randomly allocated]) and the self-esteem of those others.

You can see what happens when self-esteem values are recalculated to take into account those each actor is connected to, in this Excel file (best viewed with the NodeXL plugin). After ten iterations, Actor 0 has the highest self-esteem, and Actor 2 has the lowest. After 20 iterations Actor 2 has the highest self-esteem and Actor 1 has the lowest. After 30 iterations Actor 1 has the highest self-esteem and Actor O has the lowest. With more and more iterations the self-esteem of the actors involved might stabilise at a particular set of values, or it might repeat past patterns already seen, or maybe not.

There are two important points to be made here. The first is the dramatic affect of introducing feedback loops, in even the simplest of models. The aggregate results are not easily predictable, but they can be simulated. The second is that nature of the impact that is seen even in this very small complex system is a matter of the time period under examination. Impact seen at iteration 10 is different from iteration 20 and different again at iteration 30. In the words of the proponents of systems perspectives on evaluation, what is seen depends on the "perspective" that is chosen (Williams and Hummelbrunner, 2009).

PS 1: Michael Woolcock has written about the need to pay more attention to a related issue, captured by the term "impact trajectory". He argues that: "...in virtually all sectors, the development community has a weak (or at best implicit or assumed) understanding of the shape of the impact trajectories associated with its projects, and even less understanding of how these trajectories vary for different kinds of project operating in different contexts, at different scales and with varying degrees of implementation effectiveness; more forcefully, I argue that the weakness of this knowledge greatly compromises our capacity to make accurate statements about project impacts, irrespective of whether they are inspired by ‘demand’ or ‘supply’ side imperatives, and even if they have been subject to the most deftly implemented randomised trial"

PS1.1: Some examples:I recall that it has been argued that there is a big impact on households when they first join savings and credit groups, but the continuing impact drops down to a much more modest level thereafter. On the other hand, the impact of girls completing primary school may be the greatest when it reaches through to the next generation, to the number of their children and their survival rates.

There is one downside to my actors' self-esteem model, which is its almost excessive sensitivity. Small changes to any of the node or link values can significantly change the longer term impacts. This is because this simple model of a social system has no buffers or "slack". Buffers could be in the form of accumulated attributes of the actors (like an individual's self-confidence arising from their lifetime experience or a firm's accumulated inventory) and also provided via the wider context (like individuals having access to a wider network of friends or firms having alternate sources of suppliers) . This model could clearly be improved upon.

PS 2: I came across this quote by Duncan Watts, in a commentary on his latest book "Everything is obvious" - "when people base their decisions in part on what other people are deciding, collective outcomes become highly unpredictable" That is exactly what is happening in the self-esteem model above.Duncan Watts has written extensively on networks.

Here below is another unidirectional causal model, available on the Donor Committee for Enterprise Development website

What I like about this example is that visitors to the website can click on the links (but not in the copy I have made above) and be taken to other pages where they will be given a detailed account of the nature of the causal processes represented by those links. This is exactly what the web was designed for. Visitors can also click on any of the boxes at the bottom and find out more about the activities that input into the whole process.

The inclusion of a feedback loop in this diagram would not be too difficult to imagine. For example, from perhaps the top box back to one of the earlier boxes e.g New firms start / register. This positive feedback loop would quickly produce escalating results further up the diagram. Ideally, we would recognise that this type of outcome (simple continuous escalation) does not fit very well with our perception of what happens in reality. That awareness would then lead to further improvements to the model, which generated more realistic behaviors.

PS 24 May 2011: In their feminist perspective on monitoring and evaluation Batliwala and Pittman have suggested that we need "to develop a “theory of constraints” to accompany our “theory of change” in any given context..." They noted that "… most tools do not allow for tracking negative change, reversals, backlash, unexpected change, and other processes that push back or shift the direction of a positive change trajectory. How do we create tools that can capture this “two steps forward, one step back” phenomenon that many activists and organizations acknowledge as a reality and in which large amounts of learning lay hidden? In women’s rights work, this is vital because as soon as advances seriously challenge patriarchal or other social power structures, there are often significant reactions and setbacks. These are not, ironically, always indicative of failure or lack of effectiveness, but exactly the opposite— this is evidence that the process was working and was creating resistance from the status quo as a result .”

But it is early days. Many development programs do not yet even have a decent unidirectional causal model of what they are trying to do. In this context, the inclusion of any sort of feedback loop would be a major improvement. As shown above, the next step that can be taken is to run simulations of those improved models by inserting numerical values in the links and functions/equations in nodes of those models. In the examples above we can see how simulations can help improve models by showing how their results do not fit with our own observations of reality. Perhaps in the future they will be seen as a useful form of pre-test, worth carrying out at the earliest stages of an evaluation.

PS 3: This blog was prompted by comments to me by Elliot Stern on the relevance of modeling and simulation to evaluation, on which I hope he has more to say.

PS 4 I am struggling through Manuel DeLanda's Philosophy and Simulation: The emergence of synthetic reason (2011), which being about simulations, relates to the contents of this post.

PS 5: I have just scanned Funnell and Roger's very useful new book, Purposeful Program Theory" and found 67 unidirectional models but only 15 models that have one or more feedback loops (that is, 23%). This is quite dissapointing. So is the advice on the use of feedback loops: "We advise against using so many feedback loops that the logic becomes meaningless. When feedback loops are incorporated, a balance needs to be struck between including all of them and (because everything is related to everything else) and capturing some important ones. Showing that everything leads to everything else can make an outcome chain very difficult to understand - what we call a spagetti junction model. Neverthless some feedback loops can be critical to the success of a program and should be included ..."p187
Given the scarcity of models with feedback links even in this book, the risk of having too many feedback loops sounds like "a problem we would like to have" And I am not sure why an excess of feedback links should be any more of a probability than an excess of forward links. The concern about the understandability of models with feedback loops is however more reasonable, for reasons I have outlined above. When you introduce feedback loops what were either simple and complicated models start to exhibit complex behavior. Welcome to something that is a bit closer to the real world.

PS6: "As change makers, we should not try to design a better world. We should make better feedback loops", the text of the last slide in Owen Barder's presentation "Development Complexity and Evolution"

PS7: In Funnell and Roger's book (page 486) they describe how the Intergovernmental Panel on Climate Change (IPCC) " recognised that controlled experimentation with the climate system in which the hypothesised agents of change are systematically varied in order to determine the climate's sensitivity to these agents...[is] clearly not possible" Different models of climate change were developed with different assumptions about possible contributing factors. "The observed patterns of warming, including greater warming over land than over the ocean, and their changes over time, are simulated only by the models that include anthropogenic forcing. No coupled global climate change model that has used natural forcing only has reproduced the continental warming trends in individual continents (except Antarctica) over the second half of the 20th centrury" (IPCC, 2001, p39)

A submission to the UK Independent Commission for Aid Impact (ICAI)

2011-03-22T12:19:00.005Z

Background:

"ICAI is holding a public consultation to understand which areas of UK overseas aid stakeholder groups and the public believe the Commission should report on in its first three years. The consultation will run for 12 weeks from the 14th January until the 7th April 2011.
Click here to respond to the consultation
If you would like to read further information on ICAI, the consultation and the aid budget, please click on ‘Consultation document’.
If you would like to consider the questions in detail before responding, please click on ‘Downloadable questions’. You will need to access the online response form to respond".

Two initial comments:

1. An online survey is a fairly narrow approach to a public consultation. There are many other options, even if the ICAI is limited to those that can take place online, rather than face to face

2. The focus of the consultation is also narrow, i.e. "which areas of UK overseas aid stakeholder groups and the public believe the Commission should report on in its first three years". Equally important is how those areas of aid should be reported on.

On widening the process of consultation

1. All pages on the ICAI website should have associated Comment facilities, which visitors can make use of. More and more websites are being constructed with blog-type features such as these, because website managers these days expect to be interacting with their audience, not simply broadcasting. Built into such a facilitwould be an assumption that there will be an ongoing process of consultation, not a once off event.

2. The raw data results of the current online survey should be made publicly available, not just summaries. This is already possible, with minimal extra work required, because the survey provider (SurveyMonkey.com) is able to provide a public link to the survey results, with multiple options regarding reading, filtering and downloading the data. The more people who can access the data, the more value that might be obtained from it. However, the most important reason for doing so is that the ICAI should be seen to be maximally transparent in its operations. Transparency will help build trust and confidence in the work and judgements of the commission.

3. Although late in the day, the ICAI should edit the Consultation page to include an invitation to people to submit their own submissions using their own words and structures.

4. The ICAI website should include an option for visitors to sign up for email notification of any changes to the website, including the main content pages and any comments made on those pages by visitors.

5. The ICAI should be open about what it will be open about. It should develop a policy on transparency and place that policy on its website. Disclosure policies are now commonplace for many large international organisations like the World Bank and IMF, and transparency in regard to international aid is high on the agenda of many governments, including the UK. Having such a policy does mean everything about the workings of the ICAI must be made public, but it would typically require a default assumption of openness along with specified procedures and conditions relating to when and where information will not be disclosed.

On widening the content of consultation

1. The ICAI should be aware, if not already, that there continues to be intense debate about the best ways of assessing the value of international aid. This debate exists because of the multiplicity of purposes behind aid programs, many and varied types of aid, the enormous diversity of contexts where it is provided, the wide range of people and organisations involved in its delivery, as well as some genuinely difficult issues of measurement and analysis. There are no simple and universally applicable solutions. Value for Money provides only a partial view of aid impact, and is only partially measurable. Randomise Control Trials (RCTs) can be useful for simple replicable interventions in comparable conditions, but many aid interventions are complex. The best immediate response in these circumstances is for the ICAI to be maximally transparent about the methods being used to assess aid interventions, and to be open to the wider debate.

A good starting point would be for the ICAI to make public: (a) the Terms of Reference for the "Contracted Out Service Provider" who will do the assessment work for the ICAI, and (b) the tendered proposal put forward by the winning bidder. Both of these documents refer to ways and means of doing the required work. It is also expected that there will be periodic reviews of the work of the winning bidder. The ToRs and reports of those reviews should also be publicly disclosed on the ICAI website. Finally, all the "evaluations, reviews and investigations" to be carried out by the winning bidder on behalf of the ICAI should be publicly disclosed, as hopefully has already been agreed.

2. It would be useful, if only to help ensure that the ICAI itself delivers Value for Money", if the ICAI could clarify not only how its role will differ from that of the DFID Evaluation Department and multi-agency initiatives like 3IE, but also how it will cooperate with them to exploit any complementarities and possible synergies in their work.Complete and utter independence could lead to wasteful duplication. In the worst case various wheels could be reinvented.

For example, the OECD's Development Assistance Committee (DAC) has over the years developed a widely agree set of evaluation criteria, however these are nowhere to be seen in the ICAI's ToR for the "Contracted Out Service Provider". Instead, Value For Money receives repeated attention, and its definition is sourced to the National Audit Office (NAO). However, the NAO is a member of the Improvement Network, which has provided a wider perspective on Value for Money. Their website notes that effectiveness is part of Value for Money, as well as efficiency and economy. Commenting on effectiveness they note:

"Effectiveness is a measure of the impact that has been achieved, which can be either quantitative or qualitative....Outcomes should be equitable across communities, so effectiveness measures should include aspects of equity, as well as quality. Sustainability is also an increasingly important aspect of effectiveness."

Sustainability is a DAC evaluation criteria which has been around for a decade or more.

------------------------------------------------------------------------------------------------

A link to this blog posting has been emailed to c-robathan@icai.independent.gov.uk.

Other readers of this blog might like to do the same, with their own views.

PS: See Alex Jacob's March 22nd submission to the ICAI: Advice for the new aid watchdog

Counter-factuals and counter-theories

2010-10-20T15:23:00.009Z

Thinking about the counter-factual means thinking about something that did not happen. So consider a project involving the provision of savings and credit services, with the expectation of reducing levels of poverty amongst the participating households. The counter-factual is the situation where the savings and credit services were not provided. This can either be imagined, or monitored through the use of a control group, which is a group of similar households in a similar context.

In the course of 20 years work on monitoring and evaluation of development aid projects I have only come across one good opportunity to analyse changes in household poverty levels through the comparison of participating and non-participating households (i.e. the so called double difference method: comparing participants and non-participants, before and after the intervention). This was in Can Loc District, Ha Tinh province, in Vietnam. In 1996 ActionAid Vietnam began a savings and credit program in Can Loc. In 1997 I helped them design and implement a baseline survey of almost 600 households, being a 10% sample of the population in three communes of Can Loc District, covering participants and non-participants in the savings and credit services (which reached about 25% of all households). This was done using the Basic Necessities Survey (BNS) , an instrument that I have described in detail elsewhere.

A few years later the responsibility for the project was handed over to a Vietnamese NGO called the Pro-Poor Centre (PPC), which had been formed by ex-Action Aid staff who used to work in Ha Tinh. They continued to manage the savings and credit program over the following years. In 2006, nine years after the baseline survey, an ex-ActionAid staff member who was now working for a foundation in Hanoi, held discussions with the PPC about doing a follow up survey of the Ha Tinh households. I was brought in to assist the re-use of the same BNS instrument as in 1996. At this stage the main interest was simply to see how much households' situations had improved over the nine year period, a period of rapid economic growth throughout much of Vietnam.

The survey went ahead, and was implemented with particular care and diligence by the PPC staff. A copy of the 2006 survey report can be found here (See pages 23-25 especially). Fortunately the PPC had carefully kept hard copy records of the 1996 baseline survey (including the sample frame) and I had also kept digital copies of the data. This meant it was possible to make a number of comparisons:

Of households poverty status in 2006 compared to 1997
Of changes in the poverty status of households who were and were not participating in the saving and credit program during these periods i.e
1. Those who had never participated
2. Those were in (in 1997) but dropped out (by 2006)
3. Those who were not in (in 1997) but joined later (before 2007)
4. Those who were always in (in 1997 and 2006)

Somewhat to my surprise, I found what seemed an ideal set of results. Poverty levels had dropped the most in the 4th group ("always members"), then almost as much in the 2nd group ("ex-members"), less in the 3rd group ("new members") and least in the 1st group ("never members"). The 3rd group might have been expected to have changed less because over the years the project had expanded its coverage to include the less poor, reaching 43% of all households by 2006.

However, the project's focus on the poorest was also a problem. The members of the savings and credit program had not been randomly chosen, so the control group was not really a control group. They were not comparable. (and I had not heard of, nor still know, how to use the propensity score matching method)

The alternative to considering the effects of a counter-factual (i.e. a non-intervention) is, I guess, what could be called “counter-theoretical” That is, an alternative theory of what has happened, with the existing intervention.

My counter-theoretical centered on the idea of dependency ratios - poor families typically have high dependency ratios (i.e. many young children, relatively few adults). As families age this ratio will change, with dependent children growing up and becoming more able bodied and able to take on workloads and or generate income. Even without the access to a savings and credit program, this demographic fact alone might have explained why the participating families did better over the nine year period. It could also explain why the 2nd group did almost as well, if they were selected on the same basis of being the poorest, but had been participants for the shorter period of time.

What I could have and should have done, was go back to the PPC and see what data they had on the family structure of the interviewed households. It is quite likely they would have the relevant data: ages of all family members, given their close involvement with the community. Unfortunately at that time there was not much interest in the impact assessment aspect of the survey, by either the foundation, the PPC or ActionAid, and their support was necessary for any further analysis. Perhaps I gave up too quickly…

Nevertheless, reflection on this experience makes me wonder how often it would be well worthwhile, in the absence of good control group data, giving more attention to identifying and testing “counter-theoreticals” about the existing intervention, as part of a more rigorous process of coming to conclusions about impacts.

PS1 3rd November 2010: I have since recalled that as part of the 2006 survey I met with the staff of ActionAid in Hanoi to explain the survey process and to solicit from them their views on the likely causes of any improvements. The attached file shows two lists, one relating to ActionAid interventions in the district, and the other relating to interventions in the same district by other organisations, including government. Micro-finance was at the top of the list of the ActionAid interventions seen as likely causes of change, but there were 7 others, as well as 12 non-ActionAid interventions that were possible causes. This raises the spectre of 12 possible alternative hypotheses, let alone various combinations of these. One approach I subsequently toyed with for generating composite predictions in this kind of multiple-location/multiple-intervention situation was the "Prediction Matrix".

PS2 3rd November 2010: The current edition of Evaluation (16(4), 2010 has an article by Nicoletta Stame, titled " What doesn’t work? Three Failures, Many Answers" which includes a section on "Rival Explanations" which I have taken the liberty of copy and pasting below:

"The link between complexity and causation has been at the centre of evaluation theory ever since and has nurtured thinking about 'plausible rival hypotheses' (Campbell, 1969). Although it was originally treated as a methodological problem of validity, it has recently been revisited from the substantive perspective of programme theory. Commenting on Campbell's interest in 'reforms', that are by definition 'complex social change', Yin contrasts two strategies of Campbell: that of the experimental design and that of using rival explanations. He concludes that the second - as Campbell himself came to admit in Campbell (199a) - is better suited to complex interventions (that are changing and multifaceted), as it is with the complex case studies that have been Yin's turf for a long times (Yin, 2000: 242). The use of rival explanations is common in other crafts journalism, detective work, forensic science and astronomy), where 'the investigator defines the most compelling explanations, tests them by fairly collecting data that can support or refute them, and - given sufficiently consistent and clear evidence - concludes that one explanation but not the others is the most acceptable' (Yin, 2000: 243). These crafts are empirical: their advantage is that while a 'whole host of societal changes may be amenable to empirical investigation', especially those where stakes are currently the highest, they are 'freed from having to impose an experimental design' ('the broader and in fact more common use of rival explanations covers real-life, not craft, rivals', Yin, 2000:248). Nonetheless, rival explanations are by no means alien to evaluation, as is shown by how Campbell himself has offered Pawson good arguments for criticizing the way systematic reviews are conducted (Pawson, 2006)."

"The problem that remains is how to identify rival explanations. From a methodological starting point, Yin says that 'evaluation literature offers virtually no guidance on how to identify and define real-life rivals'. He proposes a typology of real-life rivals, that can variously relate to targeted interventions, to implementation, to theory to external conditions; and proposes examples of how to deal with them taken from such fields as decline in crime rates, support for industrial development, technological innovations, etc. However, Yin appears to overlook something that had indeed fascinated theory-based evaluation since its first appearance: the possible existence of different theories to explain the working of a programme, and the need to choose among them in order to test them. And - as Patton (1989: 377) has advised - it should be noted that in this way it would be possible to engage stakeholders in conceptualizing their own programme’s theories of action. Nevertheless, Yin’s contribution in its explicitness and methodological “correctness” is an important step forward."

"Weiss responded to Win’s provocative stance. In an article entitled “What to do until the random assigners come”, she locates Yin’s contribution as the next step beyond Campbell’s ideas about plausible rival hypotheses: “where Campbell focused primarily on rival explanations stemming from methodological artifacts, Yin proposes to identify substantive rival explanations” (Weiss, 2002: 217). She describes the process whereby the evaluator “looks around and collects whatever information and qualitative data are relevant to the issue at hand” (2002: 219), in order to see “whether any [other factor, such as other programs or policies, environmental conditions, social and cultural conditions] could have brought about the kinds of outcomes that the target program was trying to affect”, thus setting up systematic inquires into the situation. Weiss concludes that alternative means to random assignment in order to solve the causality dilemma can be a “a combination of Theory-Based Evaluation and Ruling-Out” (the rival explanation)."

I recommend the whole article...

PS3 6th December. On re-reading this post, especially Nicoletta's quote, I wondered about the potential usefullness of the "Evolving Storylines" method I developed some years ago. It could be used as a means of developing a small range of alternative histories of a project, that could then each be subject to some testing (by focusing on the most vulnerable point in each story)

Do we need a Minimum Level of Failure (MLF)?

2010-10-05T20:58:00.013Z

This week I am attending the 2010 European Evaluation Society conference in Prague. Today I have been involved in a number of interesting discussions, including how to commission better evaluations and the potential and perils of randomised control trials (RCTs). This has prompted me to resurrect an idea I have previously raised partly in jest, but which I now think deserves more serious consideration.

Background first: RCTs have been promoted as an important means of improving the effectiveness of development aid projects. But there are also concerns that RCTs will become a dominating orthodoxy, driving out the use of other approaches to impact assessment, and in the worst case, discouraging investment in development projects which are not evaluable through the use of RCTs.

In my PhD thesis many years ago I looked at organisational learning through the lense of an evolutionary epistemology. That school of thought sees evolution (through the re-iteration of variation, selection and retention) as a kind of learning process, and human learning as a sub-set of that process. As I explain below, that view of the process of learning has some relevance to the current debate on how to improve aid effectiveness. It is also worth acknowledging the results of that process - evolution has been very effective in developing some extremely complex and sophisticated lifeforms, against which intentionally designed aid projects pale in comparison.

The point to be made: A common misconception is evolution is about the “survival of the fittest”. In fact this phrase, coined by Herbert Spencer, is significantly misleading. Biological evolution is NOT about the survival of the fittest, but the non-survival of the least fit. This process leaves room for some diversity amongst those that survive, and it is this diversity that enables further evolution. The lesson here is that the process of evolution is not about picking winners according to some global standard of fitness, but about culling of failures based on their lack of fitness to local circumstances.

This leads me to my own “modest proposal” for another route to improved aid effectiveness, which is an alternative to the widespread use of RCTs and the replication of the kinds of projects found to be effective via that means. This would be to build a widening consensus about the need for a defined “Minimum Level of Failure” (MLF) within the portfolio of activities funded or implemented by aid agencies. A MLF could be something like a 10% of projects by value. Participating agencies would committ to publicly declaring this proportion of their projects as failed. Each of these agencies would also need to show: (a) how in their particular operating context they have defined these as failures, and (b) what steps they will take to avoid the replication of these failures in the future. There would be no need for a global consensus on evaluation methods, or a hegemony of methods arising through less democratic processes. PS: Using the current M&E terminology, the consensus would need to be on the desired outcomes, not on the activities needed to achieve them.

I can of course anticipate, if not already hear, some protests about how unrealistic this proposal is. Let us hear these protests, especially in public. Any agency that did so would probably be implying, if not explicitly arguing, that such a failure rate would be unacceptable, because public monies and poor people’s lives are at stake. However making such a de facto claim of a 90%+ rate of success would be a seriously high risk activity, becaus it would be very vulnerable to disproof, probably through journalistic inquiry alone. For anyone involved with development aid programmes, a brief moment’s reflection would suggest that the reality of aid effectiveness is very different, and that a 10% failure rate is probably way too optimistic and in real life failures are much more common.

Perhaps the protesting agencies might be better advised to consider the upside of a achieving a minimum level of failure. If taken seriously establishing a norm of a minimal level of failure could help get the public at large, along with journalists and politicians, past the shock-horror of failure itself and into the more interesting territory of why some projects fail. It could also help raise the level of risk tolerance, and enable the exploration of more innovative approaches to the uses of aid. Both of these developments would be in addition to a progressive improvement on the average performance of development projects resulting from a periodic culling of the worst performers.

It is possible that advocates of specific methods like RCTs (as the route to improved aid effectiveness) might also have some criticisms of the MLF proposal. They could argue that these methods will generate plenty of evidence of what does not work, and perhaps that evidence should be privileged. But the problem with this method-led solution is that there is already a body of evidence from a number of fields of scientific research that negative findings are widely under-reported. People like to publish positive findings. This may not be a big risk while RCTs are funded by one or two major actors, but it will become a systemic risk as the number of actors involved increases. There needs to be an explicit and public focus on failure.

Actual data on failure rates

PS: 15th October 2010: Four days ago I posted below some information on the success and failure rates of DFID projects. I have re-stated and re-edited that information here with additional comments:

There is some interesting data on failure within the DFID system, most notably the most recent review of Project Completion Reports (PCRs), undertaken in 2005. See the “An Analysis of Projects and programmes in Prism 2000-2005”report available on the DFID website. The percentage (68%) of projects “defined as ‘completely’ or ‘largely’ achieving their Goals (Rated 1 or 2)” was given at the beginning of the Executive Summary, but information about failures was less prominent. Under section “8. Lessons from Project Failures” on page 61 it is stated “There are only 23 projects [out of 453] within the sample that are rated as failing to meet their objectives (i.e. 4 or 5) and which have significant lessons” (italics added). This is equivalent to about 5% of the sampled projects.

More importantly are the 20% or so rated 3 = Likely to be partly achieved (see page 64). It could be argued that those with a rating of 3 should also be included as failures, since their objectives are only likely to be partly achieved, versus largely achieved in the case of rating 2. In other words a successful project should be defined as one likely to achieve more than 50% of its Output and Purpose objectives. Others are failures. This interpretation seems to be supported by a comment sent to me (whose author will remain anonymous): " "My understanding is that projects with scores of less than 2 are under real pressure and maybe quickly closed down unless they improve rapidly. I have certainly "felt the pressure" from projects to score them 2 rather than 3. That said I have not buckled to the pressure!"

I think the fact that DFID at least has a performance scoring system (for all its faults), that it has done this analysis of the project scores, and that it has made the results public, probably puts it well ahead of many other aid agencies. I would like to hear about any other agencies who have done anything like this, along with comments on the strengths and weaknesses of what they have done. I would also like to see DFID repeat the 2005 exercise at the end of this year, this time with more discussion on the projects rated 3 = Likely to be partly achieved, and what subsequently happened to these projects.

PS 2nd November 2010: See Lawrence Hadad's reference here to the same DFID set of statistics here, recently quoted/misused on the One Show

PS 3rd November 2010: Thanks to Yu-Lan van Alphen, Programmamanager, Stichting DOEN, Amsterdam, for this book reference: Kathryn Schulz – "Being Wrong", reviewed in the NYT. It sounds like a good read.

PS 14th February 2011: Computer programs are intolerant of programming errors. So, computer programmers tried to avoid them at all costs, not always successfully. Doing so becomes a much bigger challenge as software grows in size and complexity. Now some programmers are trying a different approach, that involves recognising that there will always be programming errors. For more, see "Let It Crash" Programming" by Craig Stunz at http://blogs.teamb.com/craigstuntz/2008/05/19/37819/

PS 15th February 2011: "Why negative studies are good for health journalism, and where to find them" "

This is a guest column by Ivan Oransky, MD, who is executive editor of Reuters Health and blogs at Embargo Watch and Retraction Watch.One of the things that makes evaluating medical evidence difficult is knowing whether what's being published actually reflects reality. Are the studies we read a good representation of scientific truth, or are they full of cherry-picked data that help sell drugs or skew policy decisions?..."

PS: 21 February 2011: See also the Admitting Failure website

PS: 23 April 2011. See today's Bad Science column in the Guardian by Ben Goldacre, titled "I foresee that nobody will do anything about this problem", on the difficulty of getting negative findings published

PS: 23 May 2011 .The above analysis of DFID project ratings focuses on the recognition of failures that have already occurred. It is also possible, and important, to take steps to ensure that failures are possible to be recognised in the first place. A project that has no clear theory of change will be difficult to evaluate and thus difficult to classify as a success or failure. The most common means of describing a development project’s theory of change is probably via a LogFrame representation. Within a reasonably well constructed LogFrame representation there is a sequence of “if…and…then…” statements, spelling out what is expected to happen as the project is implemented and takes effect. While there may be positive developments in a project’s Goal level indicators, there also needs to be associated evidence that the expected chain of causation leading to that Goal has also taken place as expected. It is not uncommon, in my experience, to find that while the expected outcomes have occurred, the outputs that were meant to contribute to those outcomes were not successfully delivered. In this situation the project cannot claim to be successful. There is however a more generic point to be made here. The more detailed a project’s ToC is the more vulnerable it will be to disproof. Any one of the many expected causal links could be found to have not worked as expected. However, if these linkages have not been disproved, then the stronger the project’s claims will be to have contributed to any expected and observed changes. Willingness to allow failure to be identified strengthens the claim of any success that is observed. This seems an important observation in the case of projects where there is no possibility of making comparisons with a control group where there was no intervention. In those circumstances a ToC should be as detailed and articulate as possible.

PS: 23 May 2011 :Articulating more disprovable theories of change may sound like a good idea, but it could be argued that this requirement risks locking aid agencies into a static view of the world they are working in, and one which is developed quite early in their intervention. In many settings, for example in humanitarian emergencies and highly politicised environments, aid agencies often have to revisit, revise and adapt their views of what is happening and how they should best respond. The best that might be expected in these circumstances is that those agencies are able to construct a detailed (and disprovable) history of what happened. This could actually produce better (i.e. more disprovable) results. There is some research evidence which shows that people find it easier to imagine events in some detail when they are situated in the past than to imagine the same kind of events taking place in the future[i].

[i] Bavelas, J.B. (1973) Effects of the temporal context of information, Psychological Reports, 35, 695-698, cited in Dance with Chance, by Makridakis, Hogarth and Gaba, 2009, page189.

Meta-narratives, evaluation and complexity

2010-08-26T21:36:00.010Z

A meta-narrative is a story about stories. Some evaluations take this form, especially those using participatory approaches to obtain qualitative data from a diversity of sources. Even more conventional expert-led evaluations have an element of storytelling to them as they attempt to weave information obtained from various sources, often opportunistically, into a coherent and plausible overall picture of what happened, and what might happen in future.

Recently I have come across two examples of evaluations that were very much about creating a story about stories. They raised interesting questions about method: how can it be done well?

Stories about Culture

The first evaluation was of a multiplicity of small arts projects in developing countries, funded by DOEN, a Dutch funding agency. Claudia Fontes used the Most Significant Change technique to elicit and analyse 95 stories from a sample of different kinds of participants in these projects. The aim was to identify what DOEN’s cultural intervention meant to the primary stakeholders. What particularly interested me was one part of the MSC process, which can be a useful step when faced with a large number of stories. This involved the participants categorising the stories into different groupings, according to their commonalities. It was from each of these groupings that the participants then went on to select, through intensive discussion, what they saw as the most significant changes of all. In one country five categories of stories were identified: Personal Development and Growth, Professional Development, Exposure, Change Of Perception And Attitude Towards Art And Artists, and Validation Of Self-Expression. Later on, at the report writing stage, Claudia looked at the contents of these groupings, especially the MSC stories within each, and produced an interpretation of how these groups of stories linked together. In other words, a meta-narrative.

“For the primary stakeholders in XXXX these categories of change relate to each other in that the personal and professional development of artists and other professionals who support the artists’ work results in a validation of the self-expression of direct (artists) and indirect (public in general) users. This process of affirmation and recovery of ownership of self-expression contributes in turn to a change in society’s perception of art and artists with the potential to make the whole cycle of change sustainable for the sector. Strategies of exposure have a key role in contributing across these changes, and towards the profiling of the sector in general” (italics added)

In commenting on the report I suggested that in future it might be possible and useful to take a participatory approach to the same task of producing a meta-narrative. Faced with the five groupings (and knowledge of their contents) each participant could be asked to identify expected causal connections between the different groupings, and give some explanation of these views. This can be done through a simple card sorting exercise. The results from multiple participants can then be aggregated, and the result will take the form of a network of relationships between groupings, some being stronger than others (stronger in the sense of more participants’ highlighting a particular causal linkage). This emergent structure can then be visualized using network software. Once visualised in this manner, the structure could be the subject of discussion, and perhaps some revisions. One important virtue of this kind of process is that it will not necessarily produce a single dominant narrative. Minority and majority views will be discernable. And using network visualization software, the potential complexity would be manageable. Network views can be filtered on multiple variables, such as strength of the causal linkages.

Stories about Conflict

The second evaluation was done by Lundy and McGovern, of Community –based approaches to Post-Conflict “Truth telling” in Northern Ireland” I was sent this and other related papers by Ken Bush, who is exploring methods for evaluating story-telling as a peace building methodology. His draft conceptual framework notes that a “survey of the literature highlighted the lack of an agreed and effective evaluation tool for story-telling in peace-building despite the near universality of the practice and the huge monetary investment by the EU and others in story-telling projects.”

Lundy and McGovern’s paper is a good read, because it explores the many important complications of storytelling in a conflicted society. Not only important issues like appropriate sampling of story tellers, but how the story telling project intentions were framed, and the how the results were presented. The primary product of the project was a publication called “Ardoyne: The Untold Truth”, containing testimonies based on 300 interviews. The purpose of Lundy and McGovern’s assessment of the project was “to assess the impacts and benefits of community based “truth telling”. This was done by interviewing 50 people from five different stakeholder groups. The results were then written up in their paper.

What we have here is daunting in its complexity: (a) There are the “original” stories, as compiled in the book, (b) then the stories of people’s reactions to these stories and how they were collected and disseminated, (c) and then the authors’ own story about how they collected these stories and their interpretation of them as a whole. And of course behind all this we have the complex (as colloquially used) context of Northern Ireland!

When reading what might be called Lundy and McGovern’s meta-meta-narrative (i.e. the interpreted results of the interviews) I looked for information on how sources were cited. These are the sorts of phrases I found: “according to respondents”, “many”, ”there was evidence”, “most”, “the vast majority”, “It was felt”, “respondents”, “in the main”, “for many”, “many people”, “there was a very strong opinion”, “it was felt”, “there was a consensus”. “for the majority of participants”, “without exception”, “many interviews”, “overwhelmingly”, “for others”, “some”, “for these respondents”, “one of the most frequently mentioned”, “it was further suggested”, “most respondents”, “the view”, “it was further suggested”, in general respondents were of the view that”, “the experience of those involved…would seem to suggest”, “some respondents”, “the overwhelming majority”, “responses from Union representatives were”, “for some”, “a representative of the community sector”, “that said, others were”, “by another interviewee”. “it was”, and “a significant section of mainly nationalist interviewees”.

I list these here with some hesitation, knowing how often during evaluations I have resorted to using the same vocabulary, when faced with making sense of many different comments by different sources, in a limited period of time. However there are important issues here, made even more important by how often we have to deal with situations like this. How people see things, like their reactions to the Ardoyn stories, matters. How many people see things in a given way matters, who those groups of people are matters, and how the views of different groups overlap also matters. In Lundy and McGovern’s paper we only get glimpses of this kind of underlying social structure. We sometimes get a sense of majority or minority and occasionally which particular group holds a view, and sometimes that a group sharing one view also thinks that…

How could it be done differently? The views of a set of respondents can be summarised in a "two-mode" matrix, with respondents listed in the rows and the descriptions of views listed in columns, and cell values indicating what is known about a person’s views on a listed issue. For example, agreement/disagreement, degree of agreement, or not known. By itself this data is not easy to analyse, other than through frequency counts (e.g. # of people supporting x view, or # views expressed by x person). But it is possible to convert this data into two different kinds of one-mode matrix, showing: (a) how different people are connected to each other (by their shared views) and, (b) how different views are connected to each other (by the same people holding those views). The networks structure of the data in these matrices can be seen and further manipulated using network visualization software

As in many evaluations, Lundy and McGovern were constrained by a confidentiality commitment. Individuals can be anonymised by being categorized into types of people, but this may have its limits if the number of respondents is small and the identity of participants is known to others (if not their specific views). This means the potential to make use of the first kind of network visualization (i.e. a) may be limited, even if the network visualization showed the relationships between types of respondents. However, the second type (i.e. b) should remain an option. To recap, this would show a network of opinions, some strongly linked to many others because they were often shared by the other respondents, others with weaker links to fewer others because they were shared with few, if any, other respondents. The next step would be the development of a narrative, commentary explaining the highlights of the network structure. This would usefully focus on the contents of the different clusters of opinions, and the nature of any bridges between them, especially of the clusters expressed contrasting views.

There are two significant hurdles in front of this approach. Typically not all respondents will express views on all topics, and the number who do will vary across topics. One option would be to filter out the views with the least number of respondents. The other, which I have never tried, would be interesting to explore. That would be to build in a supplementary question in interviews, along the lines of “…and how many people do you think would feel the same way as you on this issue?”. Their answers would be important in themselves, possibly affecting how the same people might act on their own views. But the same answers could also provide a weighting mechanism for views in an otherwise small sub-sample.

The second hurdle is that the network description of the relationships between the participants views is a snapshot in time. But an evaluation usually requires comparison, with a prior state. This is a problem if the questions asked by Lundy and McGovern were about current opinions. but if they were about changes in people's views it would not be.

Lets return to the layer below, the stories collected in the original “Ardoyne: The Untold Truth” publications. Stories beget stories. The telling of one can prompt the telling of another. If stories can be seen as linked in this way, then as the number number of stories recounted grows we could end up with a network of stories. Some stories in that network may be told more often than others, because they are connected to many others, in the minds of the storytellers. These stories might be what complexity science people call "attractors" Although storytellers may start off telling various different stories, their is a likelihood many of them will end up telling this particular story, because of its connectedness, its position in the network. If these stories are negative, in the sense of provoking antipathy towards others in the same community, then this type of structure may be of concern. Ideally the attractors, the highly connected stories in the network would be positive stories, encouraging peace and cooperation with others. This network structure of stories could be explored by an evaluator asking questions like "What other stories does this story most remind you off? or, "Which of these stories does that story most remind you of?" Or versions thereof. When comparing changes over time the evaluator's focus would then be on the changing contents of the strongly connected versus weakly connected stories.

In this discussion above I have outlined how a network approach could help us construct various types of aggregated (network) views of multiple stories. Because they are built up out of the views of individuals, it would be possible to see where there were varying degrees of agreement within those structures. They would not be biased towards a single (excluding others) narrative, a concern of many people using story-telling approaches including some of the originators of the term meta-narrative

And complex histories

My final comments relate to another form of story-telling, that is grand narrative as done by historians. Yesterday I read with interest Niall Ferguson’s Complexity and Collapse: Empires on the Edge of Chaos(originally in Foreign Affairs). In this article Niall describes the ways some historians have sought to explain the rise and fall in empires, in terms of sequences of events taking place over long periods. In his view they suffer from what Nassim Taleb calls "the narrative fallacy": they construct psychologically satisfying stories on the principle of post hoc, ergo propter hoc ("after this, therefore because of this”). That is, the propensity to over-explain major historical events, to create a long and coherent story where in fact there was none. His alternate view is couched in terms of complexity theory ideas. Given the complexity of modern societies “In reality, the proximate triggers of a crisis are often sufficient to explain the sudden shift from a good equilibrium to a bad mess.” He then qualifies the notion of equilibrium: “a complex economy is characterized by the interaction of dispersed agents, a lack of central control, multiple levels of organization, continual adaptation, incessant creation of new market niches, and the absence of general equilibrium.” Within those systems small changes can have catastrophic (i.e. non-linear) effects, because of the nature of the connectivity involved. Ferguson then goes onto recount examples of the rapidity of decline in some major empires.

One point which he does not make, but which I think is implicit in his discription of how change can happen in complex systems is that more than one type of small change can trigger the same kind of large scale change. Consider the assissination of Archduke Franz Ferdinand of Austria in Sarajevo in June 1914. Would World War 1 not have happened if that event took place? Not speaking as a historian..my guess is that there are quiet a few other events that could have triggered the start of a war thereafter.

Niall Ferguson complexity based view is in a sense a technocrat’s objection to grand narratives, but perhaps also another kind of grand narrative in its own right. Nevertheless his view does seem to have practical relevance to the writing of evaluation stories: it highlights the need for caution about excessive internal coherence in any story of change and its causes. A network view of causal relationships between types of events, constructed by participants with differing views, might help mitigate against this risk, when it needs to be reduced to a text description.

PS: "In recent years, however, advancements in cognitive neuroscience have suggested that memories unfold across multiple areas of the cortex simultaneously, like a richly interconnected network of stories, rather than an archive of static files." in The Fully Immersive Mind of Oliver Sacks

PS 25 October 2010. Please also see Networks of self-categorised stories

Cynefin Framework versus Stacey Matrix versus network perspectives

2010-08-20T15:51:00.055Z

Cynefin

Lots of people seem to like the Cynefin Framework. Jess Dart and Patricia Rogers are some of my friends and colleagues of mine who have expressed a liking for it. It was one of the subjects of discussion in the recent Evaluation Revisited conference in Utrecht in May. Why don’t I like it? There are three reasons...

Usually matrix classifications of possible states are based on the intersection of two dimensions. They can provide good value because combining two dimensions to generate four (or more) possible states is a compact and efficient way of describing things. Matrix classifications have parsimony.

But whenever I look at descriptions of the Cynefin Framework I can never see, or identify, what the two dimensions are which give the framework its 2 x 2 structure, and from which the four states are generated. If they were more evident I might be able to use them to identify which of the four states best described the particular conditions I was facing at a given time. But up to now I just have to make a best guess, based on the description of each state. PS: I have been told by someone recently that Dave Snowden says this is not a 2x2 matrix, but if so, why is presented like one?

My second concern is the nature of the connection between this fourfold classification and other research on complexity, beyond the field of management studies and consultancy work. IMHO, I don’t think there is much in the way of a theoretical or empirical basis for it, especially when Dave’s fifth state of “disorder”, is placed in the centre. This may be the reason why the two axes of the matrix I mentioned above have not been specified, ...because they have not yet been found.

My third concern is that I don’t think the fourfold classification has much discriminatory power. Most the situations I face, as an evaluator, could probably be described as complex. I don’t see many really chaotic ones, like gyrating stock markets or changeable weather patterns, nor do I see many that could be described as simple, or just complicated. Except perhaps when dealing with single person’s task, not involving interactions with others. Given the prevalence of complex situations, I would prefer to see a matrix that helped me discriminate between different forms of complexity, and their possible consequences.

Stacey

This brings me to Stacey's matrix, which does have two identifiable dimensions shown above: certainty (i.e. the predictability of events) and the degree of agreement over those events. Years before I had heard of "Stacey's matrix"" I had found the same kind of 2 x 2 matrix a useful means of describing four different kinds of possible development outcomes which had different implications for what sort of M&E tools would be most relevant. For example, by definition you cannot use predefined indicators to monitor unpredictable outcomes (regardless of whether we agree or disagree on their significance). However methods like MSC can be used to monitor these kinds of change. And a good case could be made for more attention to the use of historian's skills, especially to respond to unexpected events that are of dispute meaning. More recently I argued that weighted checklists are probably the most suitable for tracking outcomes that are predictable but where there is not necessary any agreement about their significance. A quote from Patton could be hijacked and used here "These distinctions help with situation recognition so that an evaluation approach can be selected that is appropriate to a particular situation and intervention, thereby increasing the likely utility -and actual use- of the evaluation" (page 85, Developmental Evaluation)

From what I have read I think Ralph Stacey also produced the following more detailed version of his matrix:

This has then been simplified by Brenda Zimmerman, as follows

In this version simple, complicated complex and anarchy (chaos) are in effect part of a continuum, involving different mixes of agreement and certainty. Interestingly, from my point of view, the category taking up the most space in the matrix is that of complexity, echoing my gut level feeling expressed above. This feeling was supported when I read Patton's three examples of simple, complicated and complex (page92, ibid), based on Zimmerman. The simple and complicated examples were both about making materials do what you wanted (cake mix and rocket components), whereas the complex example was about child rearing i.e. getting people to do what you wanted. More interesting still, the complex example was raising a couple of children in family, in other words a small group of people.So anything involving more people is probably going to be a whole lot more complex. PS: And interestingly along the same lines, the difference between simple and complicated was a physical task involving one person (following a recipe) and one involving large numbers of people (sending a rocket into space)

Another take on this is given by Chris Rodgers comments on Stacey’s views:

“Although the framework, which Stacey had developed in the mid-1990s, regularly crops up in blogs, on websites and during presentations, he no longer sees it as valid and useful. His comment explains why this is the case, and the implications that this has for his current view of complexity and organizational dynamics. In essence, he argues that

life is complex all the time, not just on those occasions which can be characterized as being “far from certainty” and “far from agreement” …
this is because change and stability are inextricably intertwined in the everyday conversational life of the organization …
which means that, even in the most ordinary of situations, something unexpected might happen that generates far-reaching and unexpected outcomes …
and so, from this perspective, there are no “levels of complexity” …
nor levels in human action that might usefully be thought of as a “system”.

Well maybe,… but this is beginning to sound a bit too much like the utterances of a Zen master to me. Like Rodgers, I hope we can still make some kind of useful distinctions re complexity.

Back to Snowden

Which brings me back to a more recent statement by Dave Snowden, which to me seems more useful than his earlier Cynefin Framework. In his presentation at the Gurteen Knowledge Cafe, in early 2009, as reported by Conrad Taylor, "Dave presented three system models: ordered, chaotic and complex. By ‘system’ he means networks that have coherence, though that need not imply sharp boundaries. ‘Agents’ are defined as anything which acts within a system. An agent could be an individual person,or a grouping; an idea can also be an agent, for example the myth-structures which largely determine how we make decisions within the communities and societies within which we live."

"Ordered systems are ones in which the actions of agents are constrained by the system, making the behavior of the agents predictable. Most management theory is predicated on this view of the organisation."

Chaotic systems are ones in which the agents are unconstrained and independent of each other. This is the domain of statistical analysis and probability. We have tended to assume that markets are chaotic; but this has been a simplistic view."

"Complex systems are ones in which the agents are lightly constrained by the system, and through their mutual interactions with each other and with the system environment, the agents also modify the system. As a result, the system and its agents ‘co-evolve’. This, in fact, is a better model for understanding markets, and organisations.”

This conceptualization is simpler (i.e. has more economy) and seems more connected with prior research on complexity. My favorite relevant quote here is Stuart Kauffman’s book: At home in the Universe: The search for the laws of complexity (p86-92) where he describes the behavior of electronic models of networks of actors (with on/off behavior states for each actor) moving from simple to complex to chaotic patterns, depending on the number of connections between them. As I read it, few connections generate ordered (stable) network behavior, many connections generate chaotic (apparently unrepeating) behavior, and medium numbers (where N actors = N connections) generate complex cyclical behavior. (See more on Boolean networks).

This relates back to conversation that I had with Dave Snowden in 2009 about the value of a network perspective on complexity, in which he said (as I remember) that relationships within networks can be seen as constraints. So, as i see it, in order to differentiate forms of complexity we should be looking at the nature of the specific networks in which actors are involved: Their number, the structure of relationships, and perhaps the extent to which the actors have own individual autonomy i.e. responses which are not specific to particular relationships (an attribute not granted to “actors” in the electronic model described).

My feeling is that with this approach it might even be possible to link this kind of analysis back to Stacey’s 2x2 matrix. Predictability might be primarily a function of connectedness, and therefore more problematic in larger networks where the number of possible connections is much higher. The possibility of agreement, Stacey’s second dimension, might be further dependent the extent to which actors’ have some individual autonomy within a given network structure.

To be continued…

PS1:Michael Quinn Patton's book on Developmental Evaluation has a whole chapter on "Distinguishing Simple, Complicated, and Complex". However, I was surprised to find that despite the book's focus on complexity, there was not a single reference in the Index to "networks". There was one example of a network model (Exhibit 5.3) , contrasted with a Linear Program Logic Model..." (Exhibit 5.2) in the chapter on Systems Thinking and Complexity Concepts. [I will elaborate further here]

Regarding the simple, complicated and complex, on page 95 Michael describes these as "sensitising concepts, not operational measurements" This worried me a bit, but it is an idea with a history (Look here for other views on this idea). But he then says "The purpose of making such distinctions is driven by the utility of situation recognition and responsiveness. For evaluation this means matching the evaluation to the nature of the situation" That makes sense to me, and is how have I tried to use the simple version of the Stacey Matrix (using dimensions only). However, Michael then goes on to provide, perhaps unintentionally, evidence of how useless these distinctions are in this respect, at least in their current form. He describes working with a group of 20 experienced teachers to design an evaluation of an innovative reading program. "They disagreed intensely about the state of knowledge concerning how children learn to read..Different preferences for evaluation flowed from different definitions of the situation. We ultimately agreed on a mixed methods design that incorporated aspects of both sets of preferences". Further on in the same chapter, Bob Williams is quoted reporting the same kind of result (i.e conflicting interpretations), in a discussion with health sector workers. PS 25/8/2010 - Perhaps I need to clarify here - in both cases participants could not agree on whether the situation under discussion was simple, complicated or complex, and thus these distinctions could not inform their choices of what to do. As I read it, in the first case the mixed method choice was a compromise, not an informed choice.

PS2: I have also just pulled Melanie Mitchell's "Complexity: A Guided Tour" off the shelf, and re-scanned her Chapter 7 on "Defining and Measuring Complexity". She notes that about 40 different measures of complexity have been proposed by different people. Her conclusion, 17 pages later, is that "The diversity of measures that have been proposed indicates that the notions of complexity that we're trying to get at have many different interacting dimensions and probably cant be captured by a single measurement scale" This is not a very helpfull conclusion. But I noticed that she does cite earlier what seem to be three categories of measures that cover many of the 40 or so measures: These are: 1. how hard the object or process is to describe?, 2. How had it is to create?, and 3. What is its degree of organisation?

PS3: I have followed up John Caddell's advice to read a blog post by Cynthia Kurtz (a co-author of the IBM Systems Journal paper on Cynefin) recalling some of the early work around the framework. In that post was the following version of the Cynefin Framework included in the oft-mentioned "The new dynamics of strategy: Sense-making in a complex and complicated world" published in the IBM SYSTEMS JOURNAL, VOL 42, NO 3, 2003.

In her explanation of the origins of this version she says it had two axes: "the degree of imposed order" and "the degree of self-organization." This I found interesting because these dimension have the potential to be measurable. If they are measureable, then the actual behavior of four identified systems could be compared. And we could then ask "Does their behavior differ in ways that have consequences for managers or evaluators?" I have previously speculated that there might be network measures that could describe these two measures: network density and network centrality. Network centrality could be the x axis, being low on the left and high on the right, and network density could be the y axis, low on the bottom and high on the top. How well the differences in these four types of network structures might capture our day-to-day notion of complexity is not yet clear to me. As mentioned way above, density does seem to be linked to differences between simple, complex and chaotic behavior. Maybe differences in centrality moderate/magnify the consequences of different levels of network density?

PLEASE NOTE: To make a Comment, or to read the Comments already made on this post, click on the Leave a Comment link below, or directly here

Evaluating a composite Theory of Change (ToC)

2010-04-25T10:29:00.003Z

One NGO I have been working with has a project that is being implemented via partner organisations in four different countries. There is one over-arching LogFrame capturing the generic ToC, but the situation in each country is quite different. So country specific LogFrames were developed to recognise that fact. However, for convenience of reporting to the back donor the progress report format has been based on the contents of the generic LogFrame. When it comes to the Mid-Term Review more attention will need to be paid to the country specific LogFrames. But then how will the four MTR results be systematically aggregated into one synthesis report?

Other colleagues of mine have had to review a funding mechanism involving 30 or more project partners. The diversity of activities on the ground there is even greater. Rather than focus on the original LogFrame that describes the purpose of the funding mechanism they got all the project staff and local partners together to construct a retrospective ToC that fitted all the funded activities. That is how they have dealt with the micro-macro compatibility issue.

How then have they evaluated the 30 projects in terms of this composite ToC? Well, because the ToC was reconstructed retrospectively there was no set of readily available monitoring data, for example as based on indicators in a LogFrame. Instead, the main source of evidence has been the qualitative data gathered from interviews with a large and diverse number of stakeholders. This approach has not met with approval in some quarters, which then prompted me to think how you could solve this problem. I should start by saying I do like the idea of a retrospectively constructed ToC, so long as there is also some accountability for its transition from any prior form such as a LogFrame.

My first thought was to treat the 30 projects as “units of evidence” and to try to classify them as having achieved, or not achieved, each stage in the ToC. The ToC was in a graphic form, with multiple events happening at different stages, and with lines representing causal links connecting the various events. The problem would then be how to classify each project. One possible means would be to get stakeholders to “success rank” the projects in relation to a given outcome event in the ToC, and then identify a cut-off point in the success ranking representing an acceptable level of achievement. This cut-off point would need to be explained and justified. To start with the focus of these success ranking exercises should be on those events most central to the ToC, i.e. those with many incoming and outgoing causal links. If there was sufficient time we would end up with a percentage (of projects) measure for each outcome, that could if necessary be weighted by scale of expenditure.

We could then turn our attention to examining what happened to the expected causal relationships between the events in the ToC. At this stage it might be possible to do simple 2 x 2 cross tabulations of the relationships between related pairs of events in the ToC, by counting numbers of projects that had achieved both of the events, achieved one but not the other, and achieved none at all. A Chi Square test could tell us how significant the observed relationship was.

There is however a potential difficulty with the next step in the process, which is to look at the larger picture of how all of the events are causally linked together to make the whole ToC work as expected. Event A may be linked to Event B by project X (and others) achieving both, and Event B may be linked to Event C by project Y (and others) achieving both. But if project X and Y are in different locations then the connection between Event A and C would seem very questionable, because the two Event Bs probably also happened in two different locations. In the worst case we could end up with a ToC where many of the individual casual links were working as expected, but where there was no evidence of the whole set of causal links working together. At the least we would have to investigate the relationships between the projects that created the larger causal pathway. In the imagined example above, we would need to look at projects X and Y and how their achievements of Event B inter-related.

This is, and sounds, complex. One means of simplification would be to identify and focus on the most important causal pathway in the ToC. In my colleagues’ ToC there were at least two major pathways and a number of minor variations. Identifying points of failure in the network of events would be one way forward. This could be done in two stages: 1. Success ranking might show that some outcome events in the ToC were not satisfactorily achieved by any projects. In that case the pathway it belonged to would be broken, 2. Chi square tests might show that although some pairs of events both took place, there was no significant association between them (in the form of the number of projects achieving both). Again the pathway they were part of would be broken.

This reflection on causal pathways make me think that the challenge of having to sort out “the attribution problem” for the final event in the ToC might be what could be called “a problem we would like to have”. It would presume that we have already established a plausible pathway of influence of our own activities. The discussion above suggests it may not be so easy in some cases.

Reflections on Dave Snowden’s presentations on sense-making and complexity

2009-12-02T12:51:00.026Z

... at the Wageningen Innovation Dialogue, 30 November -1^st December 2009

From my point of view, one of the most interesting and important challenges is how to create useful representations of large, complex, dynamic structures, especially as seen by participants in those structures. For example, multi stakeholder processes in operation at national and international levels. Behind this view is an assumption, that if we have better representations then this will provide us with more informed choices about how to respond to that complexity. Note that the key word here is respond to, not manage. The scale of ambition is more modest. Management of complexity only seems feasible when it is on a small scale, such as the children’s play group example cited by Dave Snowden (DS).

I have had a long standing interest in one particular set of tools that can be used for producing representations of complex structures. These are social network analysis (SNA) methods and associated software. During the workshop Steve Waddell provided a good introduction to SNA and related tools.

DS’s presentations on the sense-making approach provided a useful complementary perspective. This was all about making use of large sets of qualitative data, of a kind that cannot be easily used by SNA tools. Many of this data was about people’s voices, values and concerns, all in the form of fairly unstructured and impromptu responses to questions asked by their peers (who were trained to do so). These are called “micro-narratives” (MNs).

DS’s sense-making process (and associated software) is innovative in at least three respects. Firstly, in terms of the huge scale. Up to 30,000 items of text collected and analysed in one application.In many cases this would be more like a census than a sample survey. I have never heard of qualitative data being collected on this scale before. Nor as promptly, including the time spent on analysis, in the case of the Pakistan example. Secondly, and related to this, is the sophistication and apparent user friendliness of the bespoke software and hardware that was used.

More interesting, and more important, was the decision to ask respondents to “self-signify” the qualitative information they had provided. This was done by asking the respondents to describe their own MNs by using two different kinds of scales, to rate the presence of different attributes already identified by the researchers as being of concern. The consequence of respondents providing this meta-data was that all the MNs could be given a location in a three dimensional space. In fact a number of different kinds of three dimensional spaces, if many self-signifiers were used. Within that space it was then possible for the researcher to look for clusters of MNs. Of special interest were clusters of MNs that were outliers, i.e. those that were not part of the centre of the overall distribution of MNs.

There are echoes here of the expectation that the collection and analysis of Most Significant Change (MSC) stories will help organisations identify “the edges of experience”, which they wanted to see more examples of in future (if positive), or less (if negative). The difference is DS's use of quantitative data to make these outliers more identifiable, in a transparent manner.

As far as I understand it, an additional purpose of using self-signifiers to identify clusters of MNs is to prevent premature completion of the process of interpretation by the researcher, and thus to strengthen the trustworthiness of the analysis that is made.

On the first day of the workshop I had two reservations about the approach that had been described. The first was about the “fitness landscape” that was drawn within the three dimensional space. How was it constructed, and why it was needed, this was unclear to me. My understanding now is that this surface is a mathematical projection from the 30,000 data points in that 3-D space (in the Pakistan example). A bit like a regression line in a 2D graph. One advantage of this constructed landscape is that it enable observers to have a clearer understanding of how these numerous MNs relate to each other on the three dimensions. When they are simply dots hanging in space this is much more difficult to do so.

I also wondered why “peak” locations were designated as peaks, and not troughs, and vice versa. This seems to be a matter of researcher choice. This seems okay, if the landscape has no more significance than a visual aid, as suggested above. But in some complexity studies peaks in landscapes are presented as unstable locations, and troughs as stable points, acting as “attractors”. Is it likely that any pole of any of the self-signifying scales will show this type of behaviour? If not, might it be better not to talk about fitness landscapes, or at least be very careful about not giving them more apparent significance than they merit? A related claim seems to have been made when DS said “Fitness landscapes show people where change is possible”. But is this really the case? I can’t see how it can be, unless desirable/undesirable attributes are built into the self-signifying scales chosen to create the 3D space. There is a risk that the technical language that is being used imputes more independent analytic capacity than the software has in reality.

The other concern I had was about who chooses the scales used to self-signify? I should say that I do think it is okay to derive these from a relevant academic field, or from the concerns of the client for the research. But might it provide an even more independent structuring of the MN data, if these scales were somehow also derived from the respondents themselves? On reflection, there seems to be no way of doing this when the sense- maker approach is applied on a large scale.

But on a much smaller scale I think there may be ways of doing this, by using a reiterated process of inquiry, rather than a once off process. I can provide an example by using data borrowed from a stakeholder consultation process held in rural Australia a few years ago. In the first stage respondents generated the equivalent of MNs. In this case they were short statements about how they expected a new fire prevention programme to help them and their community. These statements were in effect informal “objectives”, written in ordinary day-to-day language, on small filing cards. In the next stage the same individual stakeholders were each asked to sort these statements into a number of groups (of their own choosing), each group describing a different kind of expectation. Each of these groups was then labelled by the respondent who created it. The data from these card-sorting exercises was then aggregated into a single cards x cards matrix, where each cell value described how often the row card had been placed in the same group as the column card.

Here the card sorting exercise was in effect another means of self-signifying. It was generating meta-data, statements (group labels) about the statements (individual expectations). Unlike the tripolar and bipolar scales used in David’s sense-making approach, it did not enable a 3D space to be generated where all the 30 statements could be given a specific location. However, the cards x cards matrix was a data set that many SNA software tools can easily use to construct a network diagram, which is a 2D presentation of complex structures. The structure that was generated is shown below. Each node is a card, each link between two cards represents the fact that those two cards were placed in the same group one or more times (shown by line thickness). Clusters of cards all linked to each other were all placed in the same group one or more times.When using one software package (Visualyzer), a “mouseover” on any node can be used to show not only the original card contents (the expectation), but also the labels of the one or more groups that the card was later placed in.In this adapted use of self-signifiers the process of grouping cards helps add additional qualitative information and meaning to that already there in the card contents.

As well as being able to identify respondent defined clusters of statements, we can also sometimes see links between these clusters. The links are like a more skeletal version of the landscape surface discussed above. The “peaks” of that landscape are the nodes connected by strong links (i.e. the two cards were placed the same groups multiple times). These can be made easier to identify by applying a filter to screen out the weaker links. This is the metaphorical equivalent of raising the sea level, and covering the lower levels of the landscape.

The virtue of this network approach to analysing MNs is its very participative nature. Its limitation is its modest scalability. The literature on sorting methods suggests an upper limit of between 50 or so cards (I will investigate this further).While this is much less than 30,000, many structured stakeholder consultation processes can involve a smaller numbers of participants than this.

Key: Numbers represent the IDs of each card. Links indicate that the two cards were placed in the same group, one or more times. Thicker links = placed in the same group more often. Yellow nodes = most conspicuous cliques of cards (all often co-occuring).This image shows the strongest links only(i.e. above the average number. The mouseover function is not available for this image copy.

My final set of comments are about some of the risks and possible limitations of DS’s sense-making approach. The first concern is about transparency of method. To newcomers, the complexity terminology that is used when introducing the method was challenging, to say the least. At worst I wonder whether it is an unnecessary obstruction, and whether a shorter route to understanding the method would exist, if less complexity sciences terminology was used. The proprietary nature of the associated software is also a related concern to me, though I have been told that there is an intention to make an open source version available. Open source means open to critique and open to improvement, through collective effort, which is what the progress of science is ideally all about. The extensive use of complexity science terms also seems to make the approach vulnerable to corruption and possible ridicule, as people decided to “pick and mix” the bits and pieces of complexity ideas they are interested in, without understanding the basics of the whole idea of complexity.

Another issue is commensurate benefits. After seeing the scale of the data gathering involved, and the sophistication of the software used, both of which are impressive, I did also wonder whether the benefits obtained from the analysis were commensurate with the costs and efforts that had been invested, at least in the examples we were told about. Other concerns are not exclusive to the sense-making approach. What about the stories not told? Perhaps with almost census like coverage of some groups of concern this is less of a concern than with other large scale ethnographic inquiries. What about unexpected stories? Is the search for outliers leading to the discovery of views which are a surprise to clients of the research, and of possible consequence to their plans on how to relate to the respondents in the future? And are these surprises enough in number, or are they dramatic enough, to counterbalance the resource invested to find them?

------------------

“At the heart of all major discoveries in the physical sciences is the discovery of novel methods of representation” Steven Toulmin

On the poverty of baselines and targets...

2009-10-30T22:16:00.040Z

I have been surprised to see how demanding DFID has become on the subject of baseline data. On page 13 of the new DFID Guidance (on using the new formatted Logical Framework) it is stated that ” All projects should have baseline data at all levels before they are approved. In exceptional circumstances, projects may be approved without baseline data at Output level..." Closer to the ground I have witnessed an UK NGO being pressed by DFID-appointed managers of a funding mechanism to deliver the required baseline data. This is despite the fact that the NGO's project will be implemented in a number of countries over a period of years, not all at once.

Meanwhile, in Uganda and Indonesia, I am watching two projects coming to an end. Both had baseline data collected shortly after they started. Neither is showing any signs of intending to do a re-survey at the end of the project period. Is anyone bothered? Not that I can see. Including DFID, who is a donor supporting one of the projects. And in both cases baseline surveys were expensive investments.To make matters worse, in one country the project performance targets were set before the baseline study, and in the other they have never really been agreed on.

I have just completed the final review of one project. We have diligently compared progress made on a set of indicators, against all the original targets. There are of course the usual problems of weak and missing data, and questionable causal links with project interventions. But what bothers me more is how outdated and ill-fitting some of these initial performance measures are. And how little justice this mode of assessment seems to be doing to what the project has been able to do since it started, especially the flexibility of its response in the face of the changing needs of the main partner organisation. Of even greater concern is the fact that this project is being implemented in a large number of districts, in a country that has been going through a significant process of decentralisation. Each district's capacities and needs are different, and not surprisingly the project's activities and results have varied from district to district. There is fact no one single project. Yet our review process, like many others, has in effect treated these district variations as "noise", obscuring what were expected to be region-wide trends over time.

I am now working on some ideas of how to do things differently in my next project review, in the same country. This time the focus will be more on internal comparisons: (a) between locations, (b) between time periods during the project period.

Why we should make economists work harder

2009-10-27T09:28:00.012Z

"Why we should make life harder for aid agencies" is the title of an article by Tim Harford ("The Undercover Economist") in last weekend's Financial Times magazine section.

I agree with the sentiment, but not with the analysis. I was expecting better, given what I have read of Tim in the past.

Tim's article starts with the problem of how can we, as individual donors, be sure that our aid goes in the right direction and have the expected impact. The next problem, as seen by Tim, is that aid agencies are bureacracies. The solution is competition via a more open market. From within this perspective recent efforts at aid "harmonisation" are viewed by Tim with suspicion, and seen as almost the equivalent to establishing a cartel.

He then asks could agencies be made to compete , not only with each other, but even with private companies, to get funding from donor organisations. And could money (or rather vouchers) be given directly to aid recipients to spend, redeemable for services provided by a range of charities and aid agencies. These ideas he seeas as "radical" and possibly "far fetched" More immediately, he suggests we could "start by asking simple questions about where aid comes from, where it goes, how effective it is and how much is lost to administration – or worse."

I hope Tim will be pleasantly surprised to find his ideas are not seen as radical or far fetched, and in fact have been in play now for quite come time. What Tim really needs to do (apart from more homework before writing articles like this) is to start questioning the assumptions behind his analysis of the nature and benefits of competition amongst aid agencies.

1. In most ordinary markets the purchaser and user are one and the same person. The purchaser and user of aid agency services are different parties, seperated by continents and cultures.

2. In between them is not a single supplier, but a large and complex international aid supply network. See my map of one of the simpler aid supply networks, in a Guardian funded development project in Uganda (map is at the end of the article)

3. The quality of the product/service being provided is much more difficult to assess than that found in many goods and services markets in the UK. Measurement of poverty reduction is a field of its own, improved governance is another order of magnitude more difficult to assess, but nevertheless a common development objective. There are some more measuable oucomes, such as those captured via the Millenium Development Goals e.g. reduced maternal mortality. But these usually require changes in the performance of institutions e.g. national health services. These sorts of change are not simple to measure, let alone achieve. Aid agencies can avoid this challenge by directly supplying health services to poor communities, but they will then fail on another performance metric: sustainability

Tim's idea of vouchers (above) could best be described as quaint. It is now common place for aid agencies in humanitarian emergencies to give cash handouts to families in need, not just vouchers. So they can buy what they need from anyone, not simply "a range of charities and aid agencies" Cash transfers are also being tested for their usefullness in development programmes, where there is no emergency present.

Competition between aid agencies is happening all over the place. DFID has, for years, invited tenders from a wide range of organisations to implement its aid programmes. See their Current Contract Opportunities page But what difference is this making, that is the question. By contracting out work to others DFID moves its own "overhead costs" off its own books, onto others. But the overheads are still there. In fact they are multiplied, because in order to win contracts multiple organsiations invest substantial amounts of time and effort into producing complex documents, but only one wins. Those loosing bids are not products that can easily be sold to other possible buyers, like unused factory stock. Instead the costs of their non-use is figured in to the subsequent bids, including those that win.

So, costs will have gone up, but what about effectiveness? If that has improved, then the increased costs would be justifiable. The problem is, as touched upon above, it is very diffifuclt to measure the effectiveness of many contracted-out projects, because of the scale and complexity of the changes they are trying to achieve.

Constructing longer term perspectives

2009-09-23T07:49:00.004Z

A few weeks ago a friend asked me for help with ideas for a presentation that needed to be made on "challenges for the international development sector..."

Not an easy task, where do you start? But I knuckled down and did some reflection. Work I am doing on DFID and AusAID funded projects in Indonesia ended up as the source of some ideas that may be useful. I have been working on these since late 2005 and the work continues until early 2010.

My short reply to my friend was as follows:

How to ensure that development interventions are designed and implemented within a long term perspective, that extends way beyond the typical 3-5 year planning cycles

There is a massive contradiction between the short term nature of project designs and what most people know about how long development can take (both technological and social).

If project planning cycles cannot be lengthened (e.g. because of goverment budget cycles and election cycles) then how can we make sure that these planning cycles are better linked up, into a more coherent longer term intervention? This is n easy task when there is constant staff turnover both within government and aid agencies. Strategy papers by themselves are not much use, because they have their own continuity problems, no new boss wants to simply say, yes, we will do more of same. Everyone wants to re-write the strategy in their own image

In Jakarta in October we will be holding an end-of-project review workshop for the DFID funded, GTZ implemented, AusAID monitored, government owned SISKES project (maternal and neonatal health). One of the two workshop objectives is

To engage participants in a longer term perspective on MNH development, that exceeds the typical 3-5 year project lifespan
By looking back, on developments since the beginning of the decade
By looking forward to up to four years in the future.
We will be including some people associated with a new AusAID MNH project in one of the 2 districts that GTZ are pulling out of. Plus a caste of thousands (well, 52 other participants so far).

One of the workshop exercises will be to engage participants in predicting trends in key service provision indicators over the next four years, based on their knowledge aquired through the SISKES project, and other sources. And then analysing the implications of these expected trends for the incoming projects, including the new AusAID MNH project

There is also a need for more connecting events at the design stage as well, where various stakeholders from prior and parallel related developments are brought in to inform planning decisions, or at least the choices to be considered. Often the consultants on design missions are about the only bridges to the past.

For other people's efforts to promote really long term thinking, see the Long Now Foundation

Bibliographic Timelines

2009-08-10T13:15:00.010Z

It is a simple idea, but one that looks useful

During a recent mid-term review of AMREF's Katine Community Partnerships Project, I started to create a bibliography of project related documents, with a difference. Normally documents listed in a bibliography are structured in alphabetical order, by the authors' name. This helps you find the document if you know the authors name, but not much more.

This time I listed all the project documents in time order, by the year and the month when they were produced, starting with the oldest. In the text of most reports referenced documents are usually referred to by their author and date, so it is still easy to find cited documents in this chronologically ordered list. The added advantage of this "bibliographic timeline" is that it also gives you (the reader and/or writer) a quick sense of the history of the project. Most document titles make some reference to the event they are describing (e.g. baseline studies, needs assessments, workplans, annual reports, etc), so by scanning down the bibliography you can quickly get a rough sense of the sequence of activities that have taken place. Even though there may be a time lag between an event and when it is documented (say in the next month).

I have attached below a graphic image of the "bibliographic timeline" that was produced this way. Click on the image to get more detail.

Comments on the draft DFID evaluation policy

2008-12-23T16:58:00.008Z

DFID and Independent Advisory Committee on Development Impact have sought public comments on two documents: the Draft Evaluation Policy and the Evaluation Topic List. More information on the public consultation process can be found at MandE NEWS

Comments can be emailed to evaluationfeedback@dfid.gov.uk Here below are two sets of comments that I have sent in:

1. The need for a meta-evaluation of the results of the decentralised evaluation policy

In the List of Potential Evaluation Topics, readers are invited to comment on “any topics you consider very important that we have not listed here”.

One gap which I noted was the lack of any reference to meta-evaluation of the many evaluation activities carried out within the country programmes.

However, the draft Evaluation Policy mentioned above makes eleven references to the role of “decentralised evaluation”. DFID’s decentralised evaluations “are those commissioned by our staff responsible for managing DFID’s programmes, policies and partnerships, normally in collaboration with their development partners”

The references to decentralised evaluations covered the following areas:
- increased use of decentralised evaluation as one of the 4 major priorities for developing the evaluation function in DFID. p.11
- sustaining a strong culture of decentralised evaluation across the Department. p.16
- strengthening its advisory and quality support role for decentralised evaluations p.17
- quality assurance of decentralised evaluations. p.4, p.16
- helping to set standards, providing support and advice, and reporting on quality. p4

But there are no references to a systematic or periodic meta-evaluation of decentralised evaluations. This seems like a major omission. Authority for evaluation has been decentralised, and advisory support and guidance will be provided, but there is no evident complementary mechanism for assessing the results.

PS: meta-evaluations are not the same as synthesis studies. A synthesis study looks at the findings across a number of evaluations, a meta-evaluation looks at the evaluation methods used by a number of evaluations. Most organisations, including DFID, already do quite a few sythesis studies.

2. The need for consultation on evaluation criteria, not just what should be evaluated

There needs to be some debate not just about what is to be evaluated, but on what criteria?

So far, during the present consultation, the question of what to evaluate has been subject of a separate DFID paper (the Evaluation Topic List) but the question of what criteria has only warranted a short section in an annex to the draft policy paper. In that annex DFID list “the internationally-agreed evaluation criteria …[that] will be applied to DFID evaluations. They appropriately note that while “It will not be appropriate to investigate every criterion in depth in every evaluation. DFID evaluators will be requested to provide an explanation of the criteria they have chosen (or not) to cover”. The listed criteria are 1. Relevance, 2. Effectiveness, 3. Effeciency, 4. Impact, 5. Sustainability, 6. Coverage, and 7. Coherence.

Elsewhere on this blog I have argued for the inclusion of two additional criteria to the traditional DAC 5 (1-5 above).These are equity and transparency

It could be argued that criteria 6 (coverage) already covers equity. However the choice of words can be important. Coverage is an apparently technical term, but equity is explicitly about a value: fairness, of process and outcome. DFID’s desire to eliminate of poverty is a statement about values. Values should be clearly stated, not hidden or assumed.

Transparency is not covered at all. Yet transparency is basic to the whole process of evaluation, especially when viewed in a wider context. Without access to information the ability of stakeholders in development programmes to evaluate performance on any of these criteria will be extremely limited. The importance of access to information was emphasised by the United Nations General Assembly in its first session in 1946, which states: “Freedom of information is a fundamental human right and … the touchstone of all the freedoms to which the UN is consecrated.” (Resolution 59)

More recently DFID was one of the founding signatories to the International Aid Transparency Initiative, publicised at the August 2008 High Level Forum on Aid Effectiveness in Accra, Ghana.

Given this recent statement of position by DFID transparency should clearly be included as an evaluation criteria on the DFID list. If this proposal raises concerns about the list becoming too lengthy, one could argue that it should certainly have higher priority than the newly proposed criteria 7 (coherence). In fact, perhaps it should be criteria number 1, ahead of relevance and all other criteria.

An aid bubble? - Interpreting aid trends

2008-07-24T13:24:00.003Z

(unattributed source)

This graph (from a DFID presentation) shows what many people have already heard, that the volume of aid given by DFID will continue to increase, but that the amount of money being spent on administering that aid will plateau. Does this divergence mean:

DFID has discovered a new means of effectively giving development aid that requires less and less administrative overhead each year?
There is a huge amount of slack capacity within DFID that can safely be pared away for years without hindering its effectiveness?
This graph is prima facie evidence of an impending aid bubble that is highly likely to burst in the next few years, as one or more mal-administered or corrupted aid programs are publicly exposed, to the discredit of both the good and the bad?
Yes, it does mean there will be more mal-administered and corrupted aid programs in the future, but not many more people will be worried about this than have been in the past?
This is a good example of where there is a pressing need for an ex-ante impact assessment of a budgeting strategy ( if ever there was one)?
The category “Admin budget” is meaningless and in fact the real costs of administering aid have not been adequately disaggregated in this graph.
Or...?

You can record your opinion, by posting a Comment below, or registering your vote on this anonymous opinion poll.

My initial preference would be for the fifth option, even though it is probably unlikely that the results would have much impact on decisions that have already been made.

But on reflection option one may not be so impossible as it seems. DFID may well give more and more of its aid through third parties (multilaterals, and international programmes of different kinds). When it does this those organisations' administration costs will not appear on the DFID books as administration costs, but as aid given. And those organisations can in turn use the same device to manage the apparent levels of their own administration costs, by funding other parties, such as national NGOs.

The cumulative outcome of this re-iterated strategy may well be very perverse, adding up to a bigger proportion of aid being spent on administration than would be the case if the orginal donor had been more directly engaged and been willing to show higher admin costs in its own budget. All this is speculative though. What it does suggest however, is the possible relevance of a "whole supply chain" approach to the evaluation of the costs of different forms of aid. Unlike private sector supply chains, the total cost of delivered aid is not evident in what the beneficiary pays for the final product.

Perhaps these issues could be pursued by the new Independent Advisory Committee on Development Impact?

Aid organisations as self-interested businesses?

2008-03-21T00:38:00.009Z

This posting has been prompted by a letter I received recently. A client I am working with (evaluating their project and that of another donor) wanted me to sign a confidentiality agreement. While it did not seem excessively restrictive, in terms of general intent it was the very opposite of what I have been trying to encourage this and other donors to do with information about their projects. Increasingly over the past few years I have been pushing for more transparency, not less. The rationale being that the whole aid process would benefit by being more accountable to the public at large, not just to donors or the project manager’s immediate partners and intended beneficiaries. Some of my clients have taken this approach seriously and used their websites to make a whole range of project documents publicly available (See G-rap and PETRRA). Others have agreed in principle but seem to have made little progress in practice.

Parallel to this effort I have been trying to persuade donors and project managers that achieving specific development objectives is not enough For example, increased levels of health service usage, or increased farmers’ incomes. It is also essential that knowledge be accumulated, and made available, about how these objectives were achieved, and what factors made the difference between higher and lower levels of achievement. Without this knowledge the existing achievements are less likely to be sustainable, and they are certainly unlikely to be replicable. Given the scale of most development problems, sustainability and replicability of achievements is absolutely essential. Measuring sustainability and replicability is not easy. But identifying the availability of relevant knowledge should be possible.

If information was made publicly available on how specific developments were achieved then a project can be considered to have created a public good, that others can use. The more usable that knowledge is, the more value that public good is. Businesses do not often do this, though putting usable knowledge in the public domain is becoming more common in the world of software and internet services[1]. Businesses usually have a commercial self interest in keeping secret the key parts of their business processes that would enable others to compete with them in providing the same goods or services. The production of public goods could therefore be seen as a way of differentiating the degree to which aid organisations (of varying kinds) are operating as self-interested businesses versus more public interested organisations. Whether they make and distribute a profit could be considered a secondary matter.

If the production of public goods is accepted as an important defining feature of good aid organisations then more attention to the quality of those goods, and how it could be improved, would be justified. Some might argue that a lot of information products produced by aid organisations are often more like advertising and public relations materials, and better described as “vapourware”[2]. One means of improving the quality of potential public goods would be increased transparency. So we can see not only the final information product (e.g. a book, web page, video, etc), but the drafts and the debate that surrounded their development, and the background data. Not simply as a final package, but during the process. The public could then become engaged, though comment and feedback, in the process of producing the public good(s). This type of semi-open production process is increasingly common in some areas of business (see "Democratizing Innovation", 2005 and Wikipedia entry). In aid organisations this approach could be realised in fairly simple forms through the use of websites to host draft documents, and the use of online open forums and email lists to promote awareness and discussion of those documents. This is not rocket science. But nor is it yet common practice on the scale it should be.

In my argument above transparency has two rationales. One is pragmatic, tranparency could help improve the knowledge that is available about how best to have an impact. On the other hand, when visibly put into practice, transparency may also function as an important signal of intentions, helping us differentiate organisations that are more public interested from those that are more self-interested.

[1] For example, in the form of open source products, free internet services and services that inter-operable with those provided by others.

[2] Software products that have a name, and promotional materials, but not much in the way of contents that will actually make them work and deliver what they promise

Social Frameworks: An improvement on the Logical Framework?

2008-02-01T11:29:00.012Z

Over the last week there have been quite a few email exchanges on the MandE NEWS email list about how to distinguish results from outcomes, results from impact, inputs from outputs, outcomes from impacts, etc. These are the various terms used to describe different levels of a Logical Framework description of a development intervention (in some of the variations of the Logical Framework used by some development agencies). This debate is not new; it comes and goes, and has appeared within most development organisations at some stage or another.

There are two causes of this confusion of nomenclature, in my view. One is that the Logical Framework describes a sequence of causally linked events happening over time. Time flows, it has no natural punctuation marks that can be used to distinguish and categorise stages of a process. It is not possible to “carve nature at the joints” when dealing with time. So any introduction of stage categories like inputs, activities, outputs, outcomes, impacts etc., is artificial, and requires a consensus amongst the users of these terms, if they are to be useful. Within organisations that can be achieved, across organisations it is usually much more difficult.

The second cause is a widespread confusion between two types of hierarchy. The Logical Framework is supposed to represent a temporal hierarchy, of events taking place through time. Here A is supposed to lead to B, which is supposed to lead to C, etc. However some organisations mix in a different kind of hierarchy, when they introduce terms like “components”, and “sub-objectives”. This is a hierarchy of inclusion, where A, B, and C are part of X, and X, Y, and Z are in turn part of some larger entity. So the upper levels of this hierarchy are not the outcomes of lower level activities, but simply wider generalisations or descriptions of types of things described at the lower levels. I have seen this sort of terminology in some UNICEF Logical Frameworks in Indonesia, mixed in with Purpose and Goal statements that are part of a temporal hierarchy.

A Social Framework?

I have been experimenting with an alternative, which does not “throw the baby out with the bathwater”. It could be called a Social Framework, rather than a Logical Framework, because it emphasises people and their relationships, rather than more abstract events and processes.

Let us start with the same tabular structure as the Logical Framework, but then introduce some significant changes. Each row of the narrative column (found on the left side of the Logical Framework) can be used to describe different types of actors (usually organisations or groups, rather than individuals). Actors in adjacent rows will be linked to each other by relationships that already exist, or which will be developed. The overall result is that the table describes a pathway of expected influence, from the actor in the bottom row up to the actor in the top row. The causal mechanisms are the relationships that link the actors. However, as in real life, this process of influence is unlikely to be one-directional. Both parties linked by a relationship may affect each other. For example a UK donor NGO may earn lessons from its southern partner, as well as being an important conduit of funding for that southern partner.

In the Katine project in Uganda this pathway consists of UK donors who fund AMREF who help develop the capacity of local organisations, who provide services to local households. You can see this "pathway to the poor" in the table below. However, you will see I have introduced an extra row, so I can differentiate between the internal workings of AMREF Katine as an organisational actor, and AMREF’s relationships with local organisations. As shown in the second table that follows the first, I could do the same with the other actors in the pathway. But one may not always want this degree of comprehensive detail. Nevertheless, note this basic point: actors and their relationships with each other are the basis of the Social Framework.

Simple version of the pathway

Actor	Local households
Actor	Local organisations
Relationship	AMREF’s relationship with local organisations
Actor	AMREF’s internal functioning
Actor	Donors

More detailed version of the pathway

Actor	Local households
Relationship	Local organisations’ relationship with local households
Actor	Local organisations’ internal functioning
Relationship	AMREF’s relationship with local organisations
Actor	AMREF’s internal functioning
Relationship	Donors’ relationships with AMREF
Actor	Donors’ internal functioning

It is not difficult to see some correspondence between these levels (especially in the first table) and the Logical Framework categories of Inputs, Activities, Outputs, Purpose and Goal. But talking in terms of specific categories of actors is much more tangible and communicable, especially across cultures. So, lets say goodbye to inputs, activities, outputs, etc, for the time being.

Moving on to the next column in the traditional Logical Framework, the Objectively Verifiable Indicators (OVIs), there is no reason why they cannot be used in this more Social Framework. Indicators could be identified for expected internal changes in each actor and for expected changes in their relationships with other adjacent actors in the Social Framework.

Moving on to the next column, the Means of Verificationn (MoV), this column function can also be retained, describing where and how information will be available about the expected changes. In addition, I suggest taking a more social view of this question. The MoV could describe who is expected to know about the changes described by the OVIs in the same row, because of their interests or responsibilities in this area. For example, the household row may have an indicator about households increased access to clean drinking water. In the OVI column in the same row, reference could be made to the Village Water Committee as a body who should know about changes of this type. Their knowledge and then their responses have implications for the sustainability of any improvements in water supply. This actor-oriented view implies the need for participatory approach, built around what people can and should be able to do in the way of monitoring. What is not needed is lists of disembodied items of information that might be found in a report or database somewhere.

Moving on to the next column, in the traditional Logical Framework we normally find Assumptions that refer to other factors that can influence the causal connection between events happening in adjacent rows. In the Social Framework I would suggest that this column describe assumptions about other actors, and the kind of influence that they are expected to have on the actor(s) described in the narrative row of this column (and vice versa).

The work of other NGOs in the same location may involve relationships with some of the actors in the pathways described in the Katine Social Framework. For example, the same government body, or the same community group. This could be flagged by a commentary in the Assumptions column This design flexibility contrasts with the rigidity of nested Logical Frameworks, where it is only possible to represent convergence of plans (Leading to pyramid like structures, with lots of things happening at the base, all converging on a few things at the top).

The Social Framework

This table below is a rough draft of what a Social Framework might look like for part of the Katine project in Uganda. This project is described in detail on the Guardian website.

Narrative description - of expected changes in a pathway of influence	Objectively Verifiable Indicators (OVIs), - evidence of expected change	Means of Verification (MoV), - who should know about the OVIs	Assumptions - about these and other actors
Expected changes in local households	e.g. indicators of access to safe drinking water, children's primary school participation, food sufficiency	e.g. village water committee, school management committee, village administration	e.g. the insurgents will not return again, force relocations and destroy property
Expected changes in local organisations’ relationship with local households	e.g. speed of repair of broken standpipes	e.g. village water committee, village administration	e.g. that local organisations will provide services equitably
Expected changes in local organisations’ internal functioning	e.g scores of weighted checklists re health clinic functioning	e.g. Health Unit Management Committee (HUMC)	e.g. that District Health Service will support implementation of HUMC recommendations
Expected changes in AMREF’s relationship with local organisations			e.g. AMREF will identify other NGOs who are also working with local organisations, cordinate plans with them and learn lessons re those groups
Expected changes in AMREF’s internal functioning			e.g. AMREF HQ will devolve right make public statements re the project
Expected changes in donors’ relationships with AMREF			e.g Existing donors will not prevent AMREF from seeking additional funding from other donors
Expected changes in donors’ internal functioning			e.g. Donors will be able to agree on desired outcomes of their relationship with AMREF

In complex development programmes people have tried to develop hierarchically nested Logical Frameworks, to show how different parts of a complex program connect to each other. But examples of these are not easy to find, despite the fact that there are many complex programmes in existence. In my experience, creating nested Logical Frameworks is not easy, and this may be the explanation for their scarcity.

Connecting up Social Frameworks to describe a more complex picture should be easier, because they have a modular structure. Each row describing an actor is in effect like a building block. These building blocks can be combined in different sequences. So, in addition to the pathway in the table above, a parallel Katine project pathway of influence could be Donors <-> AMREF <-> Ministry of Health <-> District Health Services <-> Local organisations <-> Local Households (actors in italics being part of other influence pathways already documented). This pathway could address the need for a parallel process of policy influencing at the national level, based on AMREF’s experience with local organisations in Katine. This pathway branches off then re-converges with the original pathway (See simple diagram version below)

As noted briefly above, each of the relationships connecting the actors in any part of the pathway is likely to involve two way communications and influence, unlike the one way causality in the Logical Framework. So messages can come back from households, via local organisations to AMREF, then go off to the Ministry of Health. Useful indicators in the AMREF row, could therefore include such developments as improved knowledge about the impact of central government policies on Katine households

PS: I have now updated the ideas describe above in a posting on MandE NEWS called The Social Framework as an alternative to the Logical Framework This is where all future developments of the idea can be found. So, please visit.

Assessing achievements in Katine, Uganda

2008-01-07T18:41:00.003Z

This weekend I will head off to Uganda for two weeks, to meet the AMREF staff working on the Katine project (See the links on the left side of this blog for more info on this project), and to see Katine sub-country itself, the place and the people. This will be the first of a series of twice-yearly visits that I will be making over the next few years. As part of the preparation for this visit on Monday this week I attended a meeting in London, to go over my Terms of Reference (ToRs) for my visit with staff from AMREF and from the Guardian.

One of the things we discussed was my request last year that AMREF develop a disclosure policy, which will spell what sorts of information they will made publicly available, and under what circumstances. Much to my surprise, that policy has already been developed and approved by the Board in November, but nobody had told me, nor had it’s existence been made public via the AMREF website. This does seem to almost defeat the purpose of the policy, which is unfortunate, since the intentions expressed in the policy do seem positive.

PS: Since that discussion a copy has now been made available on the AMREF website. My questions to you, the reader, are: What do you think of it? How could it be improved? For comparison, here is a similar sort of policy developed by ActionAid.

In the same meeting we also discussed my visit schedule in Uganda. My draft ToRs are here. As you can probably see, the list of things to do is quite long, probably too long to complete in this visit. So my first meeting with AMREF in Uganda will have to focus on prioritising these tasks. Top of my own to-do list is to meet all the AMREF staff in Katine, find out about their various roles, and to talk about their expectations about my role as the external evaluator - what they would and would not like to me doing. I will be bring along all the comments made so far by participants in an online survey of people’s views on this subject, which you can find here online. So far this online survey has focused on a limited number of stakeholders: the staff of AMREF, Guardian and Barclays. But I hope to open it up to wider public participation on return from Uganda. Please feel free to add you views on this subject right now, by commenting on this blog below.

As well as the tasks listed in my Terms of Reference there are many other questions I would like to explore during my visit. Most of these have been prompted by my reading of AMREF’s project documents over the last month, and by reading the Guardian Katine blogs. Here are some of them:

People’s participation: What did the community needs assessments find out about the existence of different community views on development needs in Katine? It is highly unlikely that in a population of 25,000 that they all had the same set of priorities. People’s views are likely to vary by gender, age, and location, at least. How have these views affected the project design?

And in AMREF’s Monitoring and Evaluation Plan for the project, what role will community groups have in monitoring and evaluation of the project? How often will their views be sought? How will those views then feed into decision making about how the project develops? [These questions relate to the equity and relevance dimension of my evaluation work]

Project strategy: Will the project be aiming to assist the whole population evenly, or will it be targeting some groups more than others? Do AMREF have enough staff and financial resources to reach the whole population? Will the various developments in water supply, health and livelihoods be focused at different target groups, or it is essential that a given group of people experience the combined impact of all these developments? How much information is available at this stage about the distribution of the population through the sub-county, and various government services? Could a map of these be made available on the Guardian Katine website, which could be continually updated and unfilled with information, as the project progresses?

Project impact: Where will the impact of the project be most visible in three years time? Will it be in changes in school attendance and completion, changes in people’s health, or changes in their livelihoods? Will the proposed baseline survey enable AMREF to track the changes that are taking place, and separate out the effects of AMREFs inputs, from the effects of other changes taking place in the society and economy? What about unexpected changes that may not have been planned for? How will they be given adequate attention? Is the monitoring and evaluation plan realistic? Is it too ambitious in terms of the information that will be collected?

Sustainability: How will the impact of the assistance provided by AMREF be sustained in the future? Will government be better able, or more willing, to take responsibility for delivering good quality health and education services?

Transparency: What mechanisms does AMREF have for transparency at the local (Katine) level, as distinct from via its website and that of the Guardian? Which of the various project documents produced so far has AMREF made publicly available? What else could be made available right now? What problems, if any, are arising because of this transparency?

If there are other issues you think I should be looking at, please add your comments below.

A network approach to the selection of "Most Significant Change" stories

2007-11-25T10:22:00.000Z

I spent yesterday in a day-long meeting with the staff of an NGO grant-making body, in Ghana. A year ago I had run a two day training workshop for their grantees on the use of the "Most Significant Change" (MSC) method of impact monitoring, a method of monitoring-without-indicators. Since then they had started to collect "Most Significant Change" stories, and they had asked for some feedback on those stories.

In yesterday's meeting, and in my meetings with other organisations in the past, concerns have been expressed about the appropriateness of a hierarchical selection process of MSC stories, when the grantees, and their local partners were all very autonomous organisations, and the last thing the grant making body wanted to do was to create, or reinforce, any view that they were all part of an organisational hierarchy, with the grant making body, and its back donors, at the top.

I explained an alternative way of structuring the selection process, that involved the parallel participation of different stakeholders groups, with a reiterated process of story selection, then feedback to the plenary meeting of all participants. After the meeting yesterday I thought it might be useful to document this alternative, and make it more widely available. So this is what is now available below, in the form of a graphic image of an Excel file. If you click on the image it will be enlarged. Or, click on the link below the image to download the actual Excel file

Your comments and suggestions are invited, please use the Comment facility on this blog.

If you have not heard about MSC before it would be worth looking at the MSC Guide first. Especially section 5 on selection.

Click on the image to make it bigger, or download the Excel file

Postscript: The Washington Post ( 31 Dec 07 online) has an interesting article about how being able to see other people's judgements affects one's own judgements. One of the authors of the study is a well know writer/researcher on networks (Duncan Watts). See also Valdis Krebs' paper "It's the [local] Conversations, Stupid: The link between social interaction and political choice"

Managing expectations about monitoring and evaluation in Katine

2007-10-20T09:03:00.000Z

Yesterday I went to an event in London, hosted by Barclays, which functioned as the official opening of the Katine project. The Guardian's Katine website went online immediately afterwards, and today's Guardian newspaper features a front page article about the Guardian's involvement in Katine, and a magazine insert giving a detailed description of Katine: the place, the people and the project.

Already some differences in expectations are evident and will need to be managed. Visits to Katine by Guardian and Barclay's staff have clearly had a psychological impact on those staff that visited, and on those they have talked to since. Others are interested to go there as well. But at the same time, AMREF staff have an understandable concern about the manageability of a stream of such visitors. How much of their staff time will be taken up with the planning and hosting of these visits, and what effect will that diversion of resources have on the implementation of the project?

My Terms of Reference (ToRs) already include a responsibility to "Assess whether the Guardian is impacting project delivery or negatively impacting the lives of the community" Already I am thinking that this responsibility needs to be amended to refer to the involvement of the Guardian and Barclays in more general terms, not just media activities.

There are some practical (M&E) steps that could be taken right now. AMREF could start to log the time spent by their staff in planning and hosting each visit by outsiders. On the Guardian and Barclays side, as I suggested to one staff member yesterday, it would be useful if those thinking about a visit could try to be as clear as possible about the objectives of their proposed visit. The nature of what would be a reasonable level of visits is also under negotiation, as part of ongoing contract discussions between AMREF, Barclays and the Guardian.

Another issue that may need to be attended to is the possible impact of the Guardian choosing to focus its media attention on Katine village, which has a population of 1500 people, although AMREF will be working with a much larger group, the 25,000 people living in the wider Katine sub-country (which Katine village is part of). It is possible, though accident and/or intention that a disproportionate amount of project resources may end up being invested in Katine village. For this and other reasons I will need to examine AMREF's plans to see how they intended to address issues of equity: who is being assisted by what project activities, and why so. This leads us into wider issues of what are the most appropriate criteria for assessing AMREF's performance, in addition to equity and effectiveness. This will be the subject of another blog posting, yet to come.

Postscript (31/10/07): I have now set up a Frequently Asked Questions(FAQs) webpage on the topic of Monitoring and Evaluating Success in Katine

Katine: an experiment in more publicly transparent aid processes

2007-10-19T08:45:00.001Z

Katine is a sub-district of Uganda (map). It is the location of an AMREF development project, funded by the Guardian, and Barclays Bank, starting this year, and scheduled to run for three years. Information about the project will be provided, and regularly updated, on a dedicated Guardian webpage

I will making a number of postings here (on Rick on the Road) and on the Guardian website, about the monitoring and evaluation (M&E) of this project.

At this early stage, there are some identifiable challenges. Some old, some new.
Old ones, which I am already familiar with, will need to be addressed by AMREF in the first instance:

Are the project objectives clear enough to be "evaluable"? Or are they just too fuzzy for anyone to judge? Right now the project staff are engaged in a process of participatory planning with people in Katine. Hopefully this will lead to some more clearly defined objectives, with identifiable and maybe even measurable outcomes, that all agree should be achieved. For example, that 95% of school age girls in the sub-district complete primary school

Amongst the many project activities (relating to education, employment, health and local governance) is there a clear sense of priorities? For example, that improvements in education are most important of all. Without this clarity, it will be hard to weigh up the different achievements and to reach a conclusion about overall success. Ideally the biggest achievements will be in the highest priority areas.

In reality there will be differing views on priorities, and even on the most important expected changes within each area (education, health, etc). Women will probably have different view to men, children will have different views to adults, poorer households will have different views to richer households, etc. Especially within a population of x,000 people. So, the third challenge will be to identify who are the different stakeholders in the project, and how their interests differ. And whose interests should the project prioritise

There are also some new issues, that I will have to address.

AMREF already has staff who are responsible for the monitoring and evaluation of the performance of its projects. But the Guardian and Barclays felt the need for an external M&E person, at least in the earliest stages of this project. The challenge for me is to make my role useful to both parties (AMREF and its two donors) but also to progressively phase out my role , as the Guardian and Barclays gain confidence in AMREF's own capacities to monitor and evaluate its own performance.

Unlike most development aid projects, this project will be in the public eye, via the Guardian, from the beginning. A Ugandan journalist will be based in the community, on a part time basis. The Guardian will be running a blog on the project for three years. There may even be a community run blog, whereby they tell the world, especially the UK, their vew of things. Where possible, project documentation will be made publicly available. All this has risks, as well as great potential for increasing public understanding about how development and aid work (and sometimes doesnt work). The second and much bigger challenge for me here is how to monitor and evaluate the impact of this public exposure

Another challenge, less threatening, will be how to best make use of this major opportunity to communicate with a large number of people. How can we get people to think about development as it happens in real life? Without drowning them in development jargon. And without reinforcing uncritical views about how easy it is to "help people" Perhaps we should start by remembering a quote from Henry Thoreau:

"If I knew for a certainty that a man was coming to my house with the conscious design of doing me good, I should run for my life...for fear that I should get some of his good done to me"

Checklists as mini theories-of-change

2007-09-01T23:32:00.000Z

During a recent evaluation of a UNICEF assisted health program in Indonesia I was given a copy of a checklist that had been designed for use in assessing the functioning of sub-district health centres in South Sulawesi. You can get a general idea of its structure from the image below. There is a list of attributes of “high performing” health centres down the left, grouped into categories, sub-categories and sub-sub-categories. Down the right side are columns, one for each health centre. Ticks are placed in each row of a column to indicate if the attribute in that row was found in that health centre. I think it is intended that if all the attributes are ticked then the health centre will be deemed to have “graduated” and no longer need to be given development assistance.

While this format has the important virtue of simplicity it does make two assumptions that may be useful to question. It appears that all the listed attributes are essential. This assumes that there is a consensus on what constitutes a “good” health centre. However, in practice, developing that consensus may be an important part of the process of developing a “good” health centre. Not only within the health centre, amongst the staff of the health centre, but also externally, amongst other organisations that the health centre has to work with (e.g. the district hospital, village health posts, and the district health office).

The second assumption is that all the attributes are of equal importance. This seems unlikely. For example, it would be widely agreed that having a supply of oxytocin (attribute 2.4.a) is much more important than “Mother's day celebration implemented each year at sub-district level” (attribute 4.2.b). Attempts to develop the capacity of the health centre will need to be guided by a clear sense of priorities, about what attributes are more important than others. The choice between organising a mothers’ day event and ensuring a supply of oxytocin could be a matter of life or death.

These two “problems” could be seen as opportunities. Attributes on the list could be weighted by asking selected stakeholders to rank the attributes in terms of their relative importance (by allocating points adding to 100 points). If there are a large number of attributes the ranking could start with the major categories, then sub-categories, then attributes within them. Importance could be defined as how much they are likely to continue to improved usage of quality services that will effect people’s health outcomes. The first set of stakeholders could be internal to the health service, and later on external stakeholders could be consulted. Attributes that were given widely different rankings would then be the focus of discussion as to why views varied so much. The assumption here is that this may lead to some convergence of views on priorities. It could also be relevant to staff training agendas. During the evaluation referred to above, we found that comparing different stakeholders ranking of the effectiveness of a number of (other) project activities generated a constructive discussion that increased both stakeholder groups’ understanding of each other, and of the issues involved.

Even when agreement is reached about appropriate weightings a question might be raised about whether this will necessarily lead to expected outcomes. Such as how women are using the health service or their behaviour after visiting the health centre. It would therefore be useful to compare the scores of different health centres and how they related to outcomes observed by those different health centres. How well do these scores predict these outcomes? If they do not, the scores could be re-calculated on the basis of a different set of weightings, to see if emphasising other attributes produced a better fit between health centre scores and observed outcomes of concern. If so, that would suggest the need for a re-orientation of priorities within the health centre. A given set of weightings is in effect the theory-of-change, and the score it generates can be treated as a prediction of an expected outcome. A series of predictions (scores from different health centres) would be needed to see how well the theory fits reality (outcomes observed by those health centres).

Incidentally: A target score on a checklist could be inserted as a single indicator in a Logical Framework, allowing a simple reference to be made to the measurement of a complex outcome. The wider use of checklist scores might help limit the use of overly simplistic indicators of progress, as seen in many Logical Frameworks.

PS: This discussion is not a criticism of the checklist as currently in use. It is an outline of what I think is some of its untapped potential.

Evolving storylines: A participatory design process?

2007-05-27T17:17:00.001Z

Some years ago...

More than a decade ago, while beginning my PhD, I experimented with the design of a process for evolving stories, through a structured participatory process. The thought was that this could lead to the development of better project designs. A project design should include a theory-of-change, and a theory-of-change when spelled out in detail can be seen as a story. But there could be many different versions of that story, some better than others. If so, then how to discover them?

One possibility was to make use of a Darwinian evolutionary process to search for solutions that have the best fit with their environment. The core of the evolutionary process is the evolutionary algorithm: the re-iteration of processes of variation, selection and retention. The intention was to design a social process that embodied these features. A similar process was later built in as a core feature of the Most Significant Changes (MSC) technique.

I tested the idea out, in a simple and light hearted way, by involving a classroom of secondary students taught by a friend of mine. The environment in which stories would have to develop and survive was that classroom, with its own culture and history. More serious applications could involve the staff of an organisation, and the environment within and around that organisation.

The process:

I gave ten of the students some small filing cards, and asked them each to write the beginning of a story on their card, about a student who left school at the end of the year. When completed, these ten cards were then posted, as a column of cards, on the left side of the blackboard, in front of the class. This provided some initial variation
I then asked the same students to read all ten cards on the board, and for each of them to identify the story beginning they most liked. This involved selection
The students were then asked to each use a second card to write a continuation of the one story beginning they most liked. These story segments were then posted next to the one story beginning they most liked. As a result, some stories beginnings gained multiple new segments, others none. This step involved retention of the selected story beginnings, and introduction of further variation.
The students were then asked to look at all the stories again, now they had been extended. I then asked them to write a third generation story segment, which they were to add to the emerging storyline they most liked so far. This process was re-iterated for four generations, until we ran out of class time. A graphic view of the results is shown below (the other being the text of the stories).

(left click to magnify image)

Each story segment is represented by one node (circle). Lines connecting the nodes, show which story segment was added to which, forming storylines. In the diagram above the story lines start from the centre and grow outwards. The color of each node represents the identity of the student who wrote that story segment. The size of each node varies according to how many "descendants" it had: how many other story segments were added to it later on. The four concentric circles in the background represent the four generations of the process. PS: Each story segment was only one to three sentences long.

The results:

In evolutionary theory success is defined in minimalist terms, as survival and proliferation. In this exercise three of the initial stories did not survive beyond the first generation (1.7, 1.8, and 1.9). Five others did survive until the fourth generation. Of these two were most prolific (1.6, 1.10), each of which had three descendants by the fourth generation.

Amongst the surviving storylines some were more collective constructions than others. Storylines 1.3 to 1.34, 1.10 to 1.39 and 1.10 to 1.38 had four different contributors (the maximum possible), whereas storylines 1.6 to 1.37 and 1.10 to 1.40 only had two.

As well as analysing the success of different storylines, we can also analyse success at the level of individual participants, using the responses of others as a measure. Individuals varied in the extent to which their story segments were selected by others, and continued by them. One participant's story segments had five continuations by others (see pale brown nodes). At the other extreme, none of the story segments of another participant (see dark green node) were continued by others. Before the exercise I had expected students to favor their own storylines. But as can be seen from the colored nodes in the diagram, this did not happen on a large scale. Some favored their own stories, but most changed storylines at one stage or another.

PS: The results of the process are also amenable to social network analysis. Participants can be seen as linked to each other through their choices of whose stories to select and add on to. It may be useful to test whether there are any coalitions at work. Either those expected prior to the exercise, or ones which were unexpected but important to know about. Within the school students exercise a social network analysis highlighted the presence of one clique of three student, where each added to each other's stories. But two of the students in this clique also added to others stories, and others added to theirs. See network diagram here.

Variations on the process

There are a number of ways in which this process could be varied:

Vary the extent to which the process facilitator tries to influence the process of evolution . The facilitator could ask all participants to start from one common story beginning in the centre. During the process the facilitators could also introduce events that all storylines must make reference to in one way or another. The facilitator could also choose to specify the some desired characteristics of the end of the story. PS: We could see the facilitator as a representative of the wider / external environment.
Run the process for a longer period. If there were ten generations, or more, it might be possible to find storylines that were built by the contributions of all ten participants. In the wider context it might be of value to find stories that have more collective ownership.
Allow participants to add two new story segments each, rather than only one. This would increase the amount of variation within the process. But it would also make the process more time consuming. It could be a useful temporary measure to create more variation amongst the stories.
Limit participation in the process to those whose (initial)storylines had survived so far. This would increase the selection pressure within the process. It could bring the process of evolution to an end (i.e one story remaining).
Magnify parts of the process. Take two consecutive segments in a story, and re-run the process to start from the first segment, with the aim of reaching the other segment by the n’th generation.
Introduce a final summary process. At the desired end time ask each participant to priority rank all the surviving storylines. These judgments could then be aggregated to provide a final score for each storyline. (Normally evolutionary processes go on and on, with different “species” emerging and dying out along the way).

How could this process be used for project development purposes?

It could be used at different stages of a project, during planning, implementation or evaluation. At the planning stage it would help think through different scenarios that the project might have to deal with. At the evaluation stage it might provide different versions of the project history, for external evaluators to look at. During implementation it could provide a mix of both scenario analysis and interpretation of history.

The mix of stakeholders involved in the process could be varied, in different ways:

The participants could be relatively homogenous (e.g. all from same organisation) or more heterogeneous (e.g. from a range of organisations), according to the amount of diversity of storylines that was desired.
The results of the process generated by one set of stakeholders (e.g. an NGO) could be subject to selection by another (e.g. the NGO's stakeholders). Using the example above, the class teacher could have indicated their preferred storyline from amongst the 10 surviving stories generated by his students.
It would also be possible to have separate roles for different stakeholders: with one group making the retention decisions (which storlines will be continued) and another making variation decisions (what new story segments to be added on to what storylines (already selected for continuation). The former could be a wide group of stakeholders, and the latter a much smaller group of project planners.

Participants could take on different roles. They could act as themselves or as representatives of specific stakeholders. Responding as individuals may allow participants to think in wider terms than when they are representing their specific stakeholder group. Stakeholder groups could participate via representatives, or as teams (each team making one collective choice about what storylines to continue, and how to do so). A team approach might promote more thought about each step in the evolving storyline, and how the stakeholder group's collective longer term interests could be best served.

At the other extreme, participants’ contributions could be anonymous (but labeled with a pseudonym). This would allow more divergent and risky contributions that might not otherwise appear.

How is this different from scenario planning?

(from Wikipedia) "Scenario development is used in policy planning, organisational development and, generally, when organisations wish to test strategies against uncertain future developments."There are many different ways in which scenario planning is done, but it appears that there are two stages, at least: (a) identification of different scenarios that are of possible concern, (b) identification of means of responding to those scenarios.

Evolving storylines is different in that both processes are interwoven and continuous. Each new story segment is a response to the previous segment, and in turn elaborates the existing scenario (story) in a particular way. It is more adaptive.

In scenario analysis it appears that scenarios are different combinations of circumstances, each of which is seen as potentially important. Such as high inflation and high unemployment. These factors are identified first, prioritised, then used to generate varous combinations. Some of these may not be able to occur together, but others that are become the scenarios. With evolving storylines there no limit on the number or kinds of elements that can be introduced into a story, but there are limits on the number of storylines that can survive.

Scenario analysis seems to be limited to a smaller number of possible outcomes than the storyline process. This may be necessary because the response process is separated from the scenario generation process.

There is also a connection to war games, as applied to the development of corporate strategy development (See Economist, May 31st 2007). These involve competing teams and the taking of turns, "allowing competitors not just to draw up their own strategies but to respond to the choices of others". Evolving storylines could take this process a step further, allowing teams to experiment with multiple parallel strategies. Sometimes a portfolio of approaches may be more useful than a single strategy, not only as a way of managing risk, but also as way of matching the diversity of contexts where an organisation is working. This is especially so for organisations working in multiple countries around the world.

Requests:

If you have any plans for testing out this process please let me know. I would be happy to provide comments and suggestions: before , during or afterwards.
I would like to develop ways of making this process work with large numbers of participants via the internet, rather than only in face to face meetings. Especially using "open source" processes that could be made freely available via Creative Commons or GNU licenses. If you have any ideas and/or capacity to help with these type of developments please left me know.

regards, rick

Prediction markets as a source of independent and continuous evaluation for development projects?

2006-12-31T17:37:00.000Z

Imagine that at the beginning of a development project, even during the planning stage, the designers identified a number of observable events, which were expected to be achieved at different points in time throughout the lifetime of the project. Some might be very immediate, such as spending 90% of the annual budget by the end of each year, while others might be much longer term in nature, such as primary school completion rates in district x exceeding 90% by 2010. Well something like this happens already in most development projects, you might say. These types of events are described in the planning documents, and associated work plans. And later on, in progress reports.

But these statements of intentions are often not very publicly accessible, and they rarely have much consequence. Cutting off funding if specific performance targets are not reached rarely happens (in my experience) and probably for good reason, it is a very crude “sledgehammer” type of response. And often the donors themselves are not independent judges, they need their projects to be seen to be succeeding, and cutting off funding is effectively a criticism of their own previous judgements, as well as the recent performance of their grantee.

There is an alternative which might be able to provide more independent and continuous assessments of project progress. These are called "prediction markets" (also sometimes called information markets, decision markets, idea futures, and virtual markets).

Prediction markets allow a group of people to express an opinion over a period of time about the probability of an event occurring. A question is posed and people buy and sell shares in stocks representing possible answers to that question. The highest priced stock at the end of a period of time is the group's prediction. (A definition provided by inklingmarkets.com)

Prediction markets have been championed in James Surowiecki’s 2004 book, “The Wisdom of Crowds”, and widely discussed and used since then (see list of links below). They have been used to successfully predict political events (election outcomes), sports events (winners), market success of commercial products, and many other types of events. Recent well known users have included Google, Yahoo, Microsoft, and HP. The most important claim that has been made is that prediction markets can generate more accurate predictions of events than individual experts or highly structured planning / design processes involving multiple specialists.

It is possible that prediction markets could also be usefully applied to development projects. Two types of benefits might arise.

Firstly, the existence of the markets might generate incentives for a wider variety and larger number of people to become engaged in a discussion about aid and development. Even prediction markets that do not involve real money bets do manage to attract large numbers of participants, who get rewarded by social recognition and self-esteem, if they win.
Secondly, those responsible for aid budgets might get more accurate information about the expected performance of their portfolio of projects than they do from those who are directly responsible for the implementation of these projects.

The challenge would be how to create incentives for project managers to publicly disclose information about their project and its performance. For example, via project websites. Rewards could be given to project managers where:

the number of participants in the prediction market was large, relative to previous comparble markets
the most favored bet was successful (predicted the observed outcome)

But this then raises the question of who would provide the rewards. Donors to the project might have the same reservations as project manegsr about disclosure of information, and reluctance to see negative predictions proved correct. The alternative would be to find independent third parties, possibly specialist NGOs, who might have an interest in promoting greater transparency by aid agencies.

Project prediction markets could have different uses at different stages. During the implementation of the project the prediction market would be providing real time feedback on expectations about whether a project was likely to succeed, that might encourage corrective behaviours by project managers. At the end of the project, when success/failure has been defined and winning bets paid off, it would be useful to compare the project manager’s own bets against the market as whole. And to analyse any discrepancies between them, and any lessons to be learnt from these.

Prediction markets can be open to the public, or internal (as used in Google, Yahoo, Microsoft, and HP). The proposal outlined above is for the use of public prediction markets in development project outcomes. But to allow and encourage project "insiders” to participate, on condition that their bets are disclosed. In the same way that directors of companies can buy and sell shares in their own company, but they are normally required to disclose these dealings.

Incidentally, the operation of prediction markets might also generate a modest income for development purposes, by using open source, proprietary or web-hosted software to host the market (see the Wikipedia listing).

Happy New Year!
Rick Davies

An experiment

Please take part in this very experimental prediction market, where the prediction concerns the achievement of one of the Millenium Development Goals (MDG). Go to
http://home.inklingmarkets.com/market/show/3166
You can leave your comments and questions in the Discussion section. Note that you have $5,000 token dollars available to spend. These are called "inkles". Bear in mind that this particular MDG prediction market is very much in the beta stage, where I expect there will be quite a few problems that will need to be sorted out.

Links:

Description & Analysis of Information Markets © 2005 by Bernd H. Ankenbrand and Caroline Rudzinski. http://www.pmcluster.com/Papers/Description%20of%20Information%20Markets%202005-12-17.pdf
Information Markets: A New Way of Making Decisions (Paperback) by Robert Hahn http://www.aei-brookings.org/publications/abstract.php?pid=1058
Putting crowd wisdom to work Posted by Bo Cowgill, Project Manager http://googleblog.blogspot.com/2005/09/putting-crowd-wisdom-to-work.html
openDemocracy markets http://www.opendemocracy.net/globalization-vision_reflections/inkling_markets_4202.jsp
Prediction markets http://en.wikipedia.org/wiki/Prediction_market
Does wisdom require markets?

Assumptions, evidence and multiple stakeholders

2006-12-17T09:49:00.000Z

Over the last few months I have been on the sidelines of a review of an NGO funding mechanism. The review report has been drafted, then re-written. But as yet, as far as I can see, three major issues have not been addressed. These are likely to be relevant to many other multi-donor NGO funding mechanisms.

Issue No. 1: The treatment of key assumptions

The first issue is core funding of NGOs (national and local). At the centre of the original project design was the belief that that provision of core funding will make an important difference to how NGOs work. The review team recognised this idea. But they did not then question or explore in any detail how the provision of core funding will lead to better development outcomes. Yet this was undoubtedly the potential killer assumption in the centre of the project design. In fact there are two linked assumptions here, that both needed examination, even if only at a desk level.

The first assumption is that core funding will increase the freedom and autonomy of NGOs. This assumption could have been explored by looking at the different NGOs that had been funded by the project, and then making some comparisons.

Firstly, by comparing NGOs where project provided core funding was a big versus small proportion of the NGO's overall budget. In the review there was no table showing such figures, though they were readily available, and though there were significant differences between NGOs in this respect. Is there any evidence of autonomy being greater where core funding was a bigger proportion of an NGO's income? Or are other factors more important in determining autonomy? An even bet, I suspect.

Secondly, by comparing the extent of the constraints imposed on NGOs by core funding mechanism, versus the constraints the NGOs experienced when using other sources of funding. Did the review team ask NGOs to compare the project (as core funder) to their other donors in terms of the constraints they imposed, and what did they find out about the differences? Complaints about funding procedures need a comparator.

The second assumption is that increased freedom and autonomy of NGOs will lead to better development outcomes.

Here it would be useful to compare the core funded NGOs’ performance against that of other NGOs who are more constrained by their donors (e.g. as a result of their project specific funding), but working on the same type of development outcomes. For example, where both NGOs work on education sector issues. At the very least it would have been possible to identify some of these cases through interviews with NGOs, and maybe even interview some of them, to at least get to the stage of developing some indicative hypotheses.

The reason for making such a comparison is that there are some good counter arguments in favour of constraint as necessary component of creativity, versus privileging freedom and autonomy. Biological evolution is the most creative process we know of, and that process works through the imposition of a very severe constraint: the need to be able to adapt to the current environment, or die. Architecture is another field where it is recognised that the presence of constraints can drive creative solutions. There have also been extensive research on the role of constraints in the fields of art and literature.

Issue No. 2: The use of evidence

The second issue was about the use of evidence. Although there had been an annual review earlier in the year, important lessons have not yet been learned from the experience. In that review there was extensive and selective use of unattributed comments by NGOs, with no information presented on how representative each of these views were. Understandably that caused major problems for the acceptance of that review. The first and second draft of the mid-term review seemed to continue that questionable tradition, albeit with a little more balance.

The alternative approach, which had been proposed before the most recent review, was that by default, all comments made by NGOs should be from identifiable sources. Exceptions could then be made where there were explainable reasons why identities had to be withheld. The assumptions behind this proposal were that:

* NGOs are mature organisations led by mature people who have a working relationship with the funder, which can withstand open expression of criticisms. Not the reverse.

* NGOs need to be confident and assertive, if they are to be effective advocates. If they cannot openly express their critical views to their own donor, how can they ever be effective advocates of critical views to less sympathetic audiences?

In contrast, the review made a brief and sympathetic reference to the earlier reviews use of evidence from NGOs, and then focused on the issue of whether results of the review interviews should be quantified or not. This was not the primary issue, and does not even need to be seen as an either/or choice. The primary issue with the both review methodologies was how transparent and trustworthy the process of data collection and analysis is. The continued selective use of unattributed comments weakened the value of both reviews.

Given there are only a small number of NGOs that had been funded it would have been quite easy to tabulate, using text not numbers, all the answers given to a number of key questions, using one table per question, and to use these in the respective relevant sections of the report.

Issue No. 3: Multi-stakeholder involvement

The third issue was multi-stakeholder involvement. The problematic nature of multi-stakeholder involvement in strategy design and evaluation was barely recognised. The project design required a common goal and convergent activities working towards that goal. Yet it involved working with NGOs who are autonomous, diverse and sometimes conflicting in their views of the world. The project's ability to find a common goal, or to mobilise people around a common goal, was in practice very very limited. Similar challenges face the whole issue of appropriate NGO representation on the project’s governing body.

Nevertheless, in this context the review proposed “Widening the dialogue on problem definition and strategy development, bringing together NGOs, government, donors and others, and using competitive funding to NGO consortia to channel demand and support the identified priorities”. And at the same time “Limit the role of the [management team] to administering grants and allied activities, avoiding other activities of a more interventionist type that might undermine the central aim”.

The review proposed that project’s strategy be defined via the proposed steering committee, and supported via “strategic issues” meetings involving a wider group of stakeholders. Although proposals were made re the inclusion of different categories of people, based largely on expertise categories, how they are chosen, and subsequently re-appointed and replaced was the more challenging question that was answered.

There is a counter argument that an independent review team could have given some attention to. That is, project’s strategy should be developed by a limited group of identifiable stakeholders with visible interests. And that NGOs using project funding should also be encouraged to seek funding from alternative sources. Overall, the project (or its donors) should be encouraging the development of plurality of funding sources, representing a diversity of strategies. Rather than trying to merge many conflicting interests within the strategy of one funding mechanism, in a non-transparent fudge that satisfies no one.

Evidence that the (development) world is getting better

2006-04-26T00:50:00.011Z

...a new approach to monitoring and evaluation ;-)

I have recently been reviewing the language we use, in the world of development aid, and come to conclusion that there is an accumulating body of evidence that the world is getting better.

Here are some examples of changes I have noticed. If you have noticed other similar changes, please post them as Comments below.

In the past we only had projects, now we have initiatives (updated 20 May 2010)

In the past we only had projects, but we have interventions (June 2011, see DFID Business Plan How to Do guidance)

In the past we had plans, but now we have strategies
In the past we did research, but now we do analytic work
In the past we interpreted data, but now we are involved in sensemaking (updated 29 May 2010)
In the past we just had stories, but now we have narratives (updated 29 May 2010)
In the past we just did monitoring and evaluation, but now we do management for development results (MDR)
In the past we were concerned about coordination, but now we are concerned about harmonisation
In the past we only wanted things to work but now we expect them to be fit-for-purpose
In the past we only had interests, but now we have passions
In the past we had problems, but now we only have issues
In the past we only had news, but now we have breaking news
In the past we were just donors, but now we are development partners
In the past we were just NGOs, but now we are Civil Society Organisations
In the past we had some concepts that were not very practically useful but now we have "sensitising concepts"
In the past we took a particular persepective... now we have an analytical lens (13 April 2011)
In the past we used to stimulate discussion, but now we open a space for a dialogue (or versions thereof) (14 April 2011)
In the past we used to ask a question or make a point in a conference, but now we make an intervention (or is this now also passe?) (14 April 2011)

Integrating funding applications and baseline surveys

2006-04-26T00:25:00.000Z

This idea falls into the category of "things I should have learned years ago!"

Over the last year or so I have been working on a number of research funding mechanisms in Ghana and Vietnam. Both involve something like the traditional two stage process of inviting simple / short Concept Notes for research, then from amongts the best of these, inviting fully developed Proposals for research. Quite a lot of information is provided by the grantee-to-be by this process, as well as by those who dont end up qualifying as grantees.

But up to now it has never occured to me that we should design this process to simultaneously gather information about the baseline status of these organisations and their activities, for subsequent monitoring and evaluation purposes. Especially information about their relationships with other actors at this early stage, which is of increasing interest to me. Instead, in one instance, we have organised a separate baseline survey some months later, involving the approved grantees only. Needless to say, this did not impress the new grantees, who had thought they had finished with form filling for the time being!

Another advantage of this approach is that by including the non-successfull applicants, we gather some wider contextual data, that will put the characteristics of the approved grantees in a broader perspective. Some of this information may reflect on the capabilty of the non-successful applicant, but other data may be more independent.

I have also been pushing a number of grant making bodies to use the application process to generate predictions of subsequent success, on a numerical scale. These predictions can later be compared to actual / perceived success, some years down the road. Not only is the correlation between the prediction and outcome of interest, so will be the positive and negative outliers (the unexpected successes and the unexpected failures). This is where case study investigations could help us learn a lot about what makes the difference between success and faulure.

The risks of big increases in aid flows to poor countries

2006-03-20T11:43:00.000Z

The latest IDS Policy Briefing (Issue 25) is titled "Increased Aid: Minimising Problems, Maximising Gains", and contains a summary of papers in a recent IDS Bulletin on the same theme. Many of the concerns raised in this Briefing relate to my own experience and echo my pre-existing concerns. In particular:

- Donors will be pre-occupied with issues of quantity, and their attention will be diverted away from the more crucial question of the quality and effectiveness of aid.

- Absorptive capacity - the ability to put aid to effective use - is already in doubt as a result of poor governance, rigid and unresponsive administrative systems, and above all, the shortage of human resources.

- Governance reforms, that might help with absorption problems, take time. Speeding them up to enable bigger aid flows, is unlikely to work

- Increased aid flows will weaken incentives for recipient government to reprioritise their existing expenditures and to improve governance

- Aid coordination efforts will not keep up with increased aid flows from multiple directions, and the burden on the host government of managing its aid will increase.

- If much of the increased aid comes in the form of loans rather than grants (from WB and others) then this will impede recipient governments' efforts to break out of indebtedness and dependence on aid.

- In smaller countries the increased volume of aid might drive up the value of the national currency making exports more expensive and undermining efforts to base growth on rising exports

In one of the (summarised) IDS papers, Ros Eyben argues that a surge in aid could magnify certain donors practices that disempower recipient governments and civil society. She is especially concerned about the potentially damaging impact of "results-based management" endorsed by donors at Monterey in 2002 as the optimal approach. Apparently because it enables donors to define what is happening, and undermines the ability of recipient governments to do so.

I am a little skeptical about the ability of any M&E method or approach to have much of an impact, but I do share her concern about the impact of large increases of foreign aid on national sovereignty and the accountability of governments to their people. Some time ago I sat in on a meeting held to discuss the findings of a report on the cost of achieving the MDGs in country x. The focus of the discussion was almost wholly on the accuracy of the calculations. What was ignored was the massive scale of the required increases and how (if delivered) they would effectively dwarf the country's own revenue sources. And even more importantly, the political implications of such a situation. What sensible government would pay much attention to its own population's expressed needs, when the bulk of its revenue was coming from external aid? No amount of "governance reform" would seem to be able to redress the effects of these perverse incentives.

I was reminded of the slogan of the American revolution "No taxation without representation", and wondered whether the truth of the reverse also needs some publicity: "No (effective) representation without taxation"

I suspect that the use of results-based management may be more a symptom, rather than a cause; a too-simple response to the perverse accountability effects of large aid flows: governments becoming less accountable to their people. RBM might be more useful if it more directly addressed the causes of this quandary: the imbalance of aid revenue versus tax revenue. It could do this by including as a key performance measure the ability of the recipient government to increase its tax revenues. Achievement could be associated with increased aid revenues, but not otherwise. The target ratio between tax and aid revenues would of course have to be identified country by country, because tax raising capacity will vary widely. Ironically this is one performance measure that the people in the recipient countries would probably not be very happy with, but which could in the longer term empower them more in relation to their governments, than any large increase in aid.

I am sure there are already many cases where increased taxation is an objective of concern to donors and host governments. But how often, if ever, have donors been willing to limit their aid flows to any percentage or multiple of recipient countries tax revenues? Here there is another set of perverse incentives at work, to do with the nature of the "aid business". That to do so would be to undermine the resource base on which aid bureacrats earn their living and where they cand find future promotion opportunities. Some will be able to create career opportunities out of a well argued need to cut aid or to limit aid growth, but not all - by definition.

Further dis-incentives would probably lay ahead, should maximum levels of aid per country ever be agreed. Donor countries would have to agree who would contribute how much to a limited pot, when in fact many were seeking to maximise their influence. This would be the ultimate "hamonisation" challenge!

Late Note (23/03/06) From "Business in Africa"

"...Taking a look at the country’s national budget, virtually all spend is draft estimates and not accounted for cent by cent. An example of this lack of budget control is highlighted in the SAIIA report, where the director of monitoring and evaluation in the ministry of economic development and planning is quoted as saying, “What we are saying is that out of the 60 billion kwacha in the 2002/2003 budget, we only know of 37 billion kwacha that was used. We don’t know what happened to the rest because of lack of data.”

Foreign support makes up close to half of Malawi’s budget, which means that it is difficult to implement policies when the budget is dependent largely on sources from outside the country. In 2000, International Monetary Fund (IMF) aid and other donor funding was suspended following high-level corruption, archaic fiscal discipline and poor governance under the Bakili Muluzi administration. The IMF and World Bank resumed aid in 2004 when the economic minister in Muluzi’s cabinet, Bingu wa Mutharika, was named president."

Okay, so running a close second is the need for a performance indicator on the availability of adequate government accounts. This would be in the interests of citizens and supporting donors. No conlfict of interest here, unless donors have privileged access to government accounts that citizens do not. Unfortunately this is not an unknown occurence.

Another part of this latest story is the stop-go nature of aid flows. In highly aid dependent countries that cannot be a good way to proceed. What sort of performance monitoring system would generate such clumsy responses? Either one that was only barely working, or being ignored by its own proponents (or both).

The "attribution problem" problem

2005-12-16T11:28:00.000Z

I have lost count of the number of times I have seen people make reference to "the attribution problem" as though doing so was a magic spell that dispelled all responsibility to do anything, or to know anything, about the wider and longer term impacts of a project. Ritualistic references to the "attribution problem" are becoming a bit of a problem.

In the worst case I have seen an internationally recognised consultancy company say that "our responsibilities stop at the Output level". And while other agencies might be less explicit, this is not an uncommon position.

This notion of responsibility is very narrow, and misconceived. It sees responsibilities in very concrete terms, delivering results in the form of goods or services provided.

A wider conception of responsibility would pay attention to something that can have wider and longer term impact. That is the generation of knowledge about what works and does not work in a given context. Not only about how to better deliver specific goods or services, but about their impact on their users, and beyond. Automatically, that means identifying and analysing the significance of other sources of influence in addition to the project intervention.

Contra to some people's impressions, this does not mean having to "prove" that the project had an impact, or working out what percentage of the outcome was attributable to the project (as one project manager recently expressed concern about). Something much more modest in scale would still be of real value. Some small and positive steps forward would include: (a) identifying differences in outcomes, within the project location [NB: Not doing a with-without trial], (b) Identifying different influences on outcomes, across those locations, (c) prioritising those influences, according to best available evidence at the time, (d) doing all the above in consultations with actors who have identifiable responsibilities for outcomes in these areas, (e) making these judgements open to wider scrutiny.

This may not seem to be very rigorous, but we need to remember our Marx (G.), who when told by a friend that "Life is difficult", replied "Compared to what?" Even if project managers choose to ignore the whole question of how their interventions are affecting longer term outcomes, other people in the locations and institutions they are working with will continue to make their own assessments (formally and informally, tacitly and expliictly). And those judgements will go on to have their own influences, for good and bad, on the sustainabilty and replicability of any changes. But in the process their influences may not be very transparent or contestable. A more deliberate, systematic and open approach to the analysis of influence might therefore be an improvement.

PS: On the analysis of internal variations in outcomes within development projects, you may be interested in the Positive Deviance initiative at http://www.positivedeviance.org/

Impact pathways and genealogies

2005-10-23T09:14:00.000Z

I have been working with three different organisations where the isssue of impact pathways has come up. Note the use of the plural: pathways. Network models of development projects allow the representation of multiple pathways of influence (whereby project activities can have an impact) whereas linear / temporal logic models are less conducive to this view. They tend to encourage a more singular vision, of an impact pathway.

In one research funding organisation there was a relative simple conception of how research would have an impact on peoples lives. It would happen by ensuring that research projects included both both researchers and practioners. Simple as it was, this was an improvement on the past, where research projects included researchers and did not think too much about practioners at all. But there was also room for improvement in this new model. For example, it might be that some research would have most of its impact through "research popularisers",who would collate and re-package research findings in user friendly forms, then communicate them on to practioners. And there may be other forms of research where the results were mainly of interest to other researchers.This might be the case with more "foundational" or "basic" research. So, there might be multiple impact pathways, including others yet not identified or imagined.

Impact pathways can not only extend out into the future, but also back into the past. All development projects have histories. Where their designs can be linked back to previous projects these histories can be seen as genealogies. The challenge, as with all genealogical research, is to find some useful historical sources.

Fortunately, the research funding organisation had an excellent database of all the research proposals it had considered, including those it had ended up funding. In each proposal the staff had added short lists of other previous research projects they had funded, which they thought were related and relevant to this project proposal. What the organisation has now is not just a list of projects, but also information about the web of expected influences between these projects, a provisional genealogy which stretches back more than ten years.

I have suggested to the organisation that this data should be analysed in two ways. Firstly, to identify those pieces of research which have been most influential over the last 10 to 15 years, simply in terms of influencing many other subsequent pieces of research. They could start by identifying which prior research projects were most frequently refered to in the lists attached to (funded) research proposals. This is very similar to citation analysis used in bibliometrics. These results would then need to be subject to some independent verification. Researchers' reports of their research findings could be re-read for evidence of the expected influence (incuding, but not only, their listed citations). They could also be contacted and interviewed.

The second purpose of a network analysis of past research would be to identify a sample of research projects that could be the focus of an ex-post evaluation. With the organisation concerned, I have argued the case for cluster evaluations, as a means of establishing how a large number of projects have contributed to their corporate objectives. But what is a cluster? A cluster could be identified through network analysis, as a groups of projects having more linkages of expected influence between themselves than they do have with other research projects around them. Network analysis software, such as UCINET, provides some simple means of identifying such clusters in large and complex networks, based on established social network analysis methods. Within those clusters it may also be of interest to examine four types of research projects, having different combinations of outwards influences (high versus low numbers of links to others) and inward influences (high versus low numbers of links from others).

Looking further afield it may of value for other grant making organisations to be more systematic about identifying linkages between the projects they have funded in the past, and those they are considering funding now. And then encouraging prospective grantees to explore those linkages, as a way of promoting inter-generational learning between development projects funded over the years.

Networks of Indicators

2005-10-22T15:48:00.000Z

A few months ago I was working with a large scale health project, that was covering multiple regions within a large country. The project design was summarised in a version of the Logical Framework. Ideally a good Logical Framework can help by providing a simplified view of the project intentions, through the use of a narrative that tells how the Activities will lead to the Outputs, via some Assumptions and Risks, and how the Outputs will lead to the Purpose level changes, via some Assumptions and Risks, and so on... Running parallel to this story will be some useful indicators, telling us when various events at each stage of the story has taken place.

That is of course in a ideal world. Often the storyline (aka the vertical logic) gets forgotten and the focus switches to the horizontal logic: ensuring there are indicators for each of the events in the narrative, and more!

Unfortunately, in this project, like many others, they had gone overboard with indicators. There were around 70 in all. Just trying to collect data on this set of indicators would be a major challenge for the project, let along analysing and making sense of all the data.

As readers of this blog may know, I am interested in network models as alternatives to the use of linear logic models (e.g. the Logical Framework) to represent development project plans, and their results. I am also interested in the use of network models as a means of complementing the use of Logical Framework. Within this health project, and its 70 indicators, there was an interesting opportunity to use a network model to complement and manage some of the weaknesses of the project's Logical Framework.

Sitting down with someone who knew more about the project than I did, we developed a simple network model of how the indicators might be expected to link up with each other. An indicator was deemed to be linked to another indicator if we thought the change that it represented could help cause the change represented by the other indicator. We drew the network using some simple network analysis software that I had at hand, called Visualyzer, but it could just have easily been done with the Draw function in Excel. I will show an "anonomised" version of the network diagram below.

When discussing the network model with the project managers we emphasised that the point behind this network analysis of project indicators was that it was the relationships between indicators that are important. To what extent did various internal Activities lead to changes in various public services provided (the Outputs)? To what extent did the provision of various Outputs affect the level of public use of those services, and their attitudes towards them (Purpose level changes)? To what extent did these various measures of public health status then related to changes in public health status (Goal level changes)?

The network model that was developed did not fall out of the sky. It was the results of some reflection on the project's "theory of change", its ideas about how things would work, how various Activities would lead to various Outputs and on to various Purpose level changes. As such it remained a theory, to be tested with data obtained through monitoring and evaluation activities. Within that network model there were some conspicuous indicators, that would deserve more monitoring and evaluation attention that others. These were indicators that (a) had an expected influence on many other indicators (e.g. govt. budget allocation), or (b) indicators that were being influenced by many other indicators (e.g. usage rates of a given health service)

The next step, on my next visit, will be to take this first rough-draft network model back to the project staff, and refine it, so it is a closer reflection of how they think the project will work. Then we will see if the same staff can identify the relationships between indicators that they think will be most critical to the project's success, and therefore most in need of close monitoring and analysis. The analysis of these critical relationships may itself not be any more sophisticated than a cross-tabulation, or graphing, of one set of indicator measure against another, with the data points reflecting different project locations.

Incidentally, the network model not only represented the complex relationships between each level of the Logical Framework, but also the complex relationships within each level of the Logical Framework. Activities happen at different times, so some can influence others, and even more so, when Activities are repeated in cycles, such as annual training events. Similarly, some Outputs can affect other Outputs, and some Purpose level changes can affect other Purpose level changes. The network model captured these, but the Logical Framework did not.

Fight institutional Alzheimers

2005-07-06T08:23:00.000Z

I have taken this headline and the following text from the POLEX: CIFOR's Forest Policy Expert Listserver run by David Kaimowitz. I am reproducing it in full because I strongly agree with David's conclusions. How many other bilateral or multilateral aid gencies have done something like this recently? If you know of others, let me know.

"They say the good thing about having Alzheimer’s disease is that you are always visiting new places and meeting new people. Many development agencies have apparently taken that to heart. Rapid staff turnover, weak efforts to save and share documents, and strong incentives to repackage old wine in new bottles keep many institutions from learning from the past.

That why it is good to see the US Agency for International Development (USAID) invest in reviewing everything they have funded related to natural forests and communities during the last twenty-five years. The result is a three-volume report called USAID’s Enduring Legacy in Natural Forests: Livelihoods, Landscapes, and Governance by a Chemonics International team led by Robert Clausen. It provides an overview and ten country studies.

Back in the 1970s, USAID’s forestry activities focused mostly on fuelwood and promoting tree planting as part of watershed management projects. Later, growing concern about deforestation made them shift towards biodiversity conservation and protected areas. After that came a move towards market-based instruments such as forest certification, ecotourism, and tapping consumer demands for non-timber forest products. Over time, they have funded more NGOs and local governments and fewer national bureaucracies. And if the report’s authors have their way, the links between natural resources, democratization, and conflict prevention will soon be high on the agenda.

Through all that time and changes, some things remained the same. For example, it is still important to invest in forests for the long-term and get the technical aspects right. You need to work with specific farms, forests, and parks, but keep your eyes on larger landscapes. If no one invests in studying and monitoring forests and their products and services, when it comes time to justify investments or make decisions the data simply won’t be there. Projects need to focus more on ethnic and cultural issues. You ignore conflicts at your own risk.

People with advanced Alzheimers can be nice and well-intentioned, but they should not be running the show. If we don’t build up our institutional memory we will keep making the same mistakes, although we may give them another name. Let’s hope other agencies follow USAID’s lead and invest in learning from their own experience."

[If you would like to receive CIFOR-POLEX in English, Spanish, French, Bahasa Indonesia, or Nihon-go (Japanese), send a message to Ketty Kustiyawati at k.kustiyawati@cgiar.org]

regards from Rick, in Cambridge

Using "modular matrices" to describe programme intentions and achievements

2005-05-23T21:38:00.000Z

There has been an interesting discussion about the pros and cons of Logical Frameworks, on the MandE NEWS mailing list. One participant has expressed concerns about the unrealistic expectations many people have about the use of the Logical Framework. We should not expect it to do everything. It is supposed to be a summary. To be read along side narrative accounts which can be as detailed as needed.

My response was to point out that there was some usable middle ground between long narrative accounts and tables that attempted to summarise a whole programe in a four by four set of cells. The middle ground is what I now call a "modular matrix approach" (MMA). Google defines "modular” as follows: “Equipment is said to be modular when it is made of "plug-in units" which can be added together to make the system larger, improve the capabilities, or expand its size”

So a Gantt chart can be seen as a modular unit, because it can build onto and extend the LogFrame. It can do this because it has one common dimension: a set of Activities. Another module that I have seen used in association with the LogFrame is a matrix of Outputs x Actors (using the outputs). Here the Outputs are the common dimension that links this matrix and a LogFrame.

In the last year or so I have experimented with a range of modules, some of which have proved more useful than others. Ideally, this development process would be a collective enterprise, such that what emerged was a public library of usable planning modules. Some, like the Logframe, would offer a very macro perspective. Others, such as an Activity x Activity module, can provide a more micro perspective on work processes within single organisations.

When developing new matrix modules I use the social network analysis convention, that cell contents should describe the relationship from the row actor to the column actor. The actors involved are listed down the left column and across the top row. In practice I also use documents (produced by actors) and events (involving actors). Such matrices allow the representation of networks of communications and influence, not just one directional chains of cause and effect.

A second important convention that I try to follow, implicit in the above description, is that the entities listed on the two axes of such matrices should be verifiable, either by interviewing them (if they are actors) or reading them (if they are documents) or reading about them (if they are events). This will then allow us to establish if the links between them were planned, and eventuated, as described. There are probably other conventions that could be developed to enure that matrix modules developed by different people are compatable, and can add value to the whole.

For some recent practical experiments along these lines see this paper. In the near future I hope to provide a comprehensive summary of this approach, in a paper provisionally titled "From Logical to Network Frameworks: A Modular Approach to Representing Theories of Change" This paper will be publicised via the Network Evaluation and MandE NEWS mailing lists.

Constructing "an auditable trail of intentions...."

2005-04-02T10:05:00.000Z

A useful report has recently been produced by ODI (Lucas, Evans, Pasteur and Lloyd, 2004)“on the current state of PRS monitoring systems. PRS are national level Poverty Reduction Strategies, promoted by multilateral and bilateral aid agencies. In that report they argue for more attention to the severe capacity constraints facing governments who are try to monitor their PRSs. Donors need to take “a less ambitious attitude as to what can be achieved and a willingness to make hard choices when prioritising activities”. Later on, in discussions about the range of indicator data that might be relevant they note that “Given scarce resources, a focus on budget allocations and expenditures may well an appropriate response, particularly if it involves effective tracking exercises with mechanisms to ensure transparency and accountability…Linking these data to a small set of basic service provision indicators that can reasonably reflect annual changes could provide a reasonable starting point in assessing if the PRS is on track.”

Meanwhile I have been working on the fringes of a PRS update process that is taking place in a west African country. While I agree with the line taken above, I am wondering now if even this approach is too ambitious! This will be the second PRS for the country I am working in. This time around the government has made it clear to ministries that their budgets will be linked in to the contents of the PRS. This seems to have had some positive effect on their levels of participation in the planning groups that are drafting sections of the PRS. By now some draft versions of the updated PRS policies have been produced, and they have been circulated for comment within a privileged circle (including donors). Some attempts have been made at explicitly prioritising policy objectives, but only in one of five policy areas. Meanwhile there is a deadline approaching at high speed, for identifying and costing the programmes that will achieve these policy objectives. This is all due by the end of this month, April 2005. Then it is expected the results will feed into a public consultation and then into the national budget process starting in June. However, as yet there is no agreed methodology for the costing process. As the deadline looms the prospects increase for a costing process that is neither systematic or transparent (aka business as usual).

If the process of constructing the costings is not visible, then it becomes very clear to identify the specific linkages between specific PRS policy objectives and specific items in the national budget. So while we can, on ODI good advice, monitor budget allocations and expenditures, what they mean in terms of the PRS policy objectives will remain an act of literary interpretation. Something that could easily be questioned.

IMF and UNDP have I think both had some involvements in costings of broad policy objectives, including the achievement of the MDGs. However, from what I can see these costings have been undertaken by consultant economists, primarily as technical exercises. But I am not sure if this is the right approach. The budgets of ministries are political resources. The alternative approach is to ask Ministries to say how they will use their budget to achieve the various PRS policy objectives, and while doing so make it clear that their performance in achieving those selected objectives with their budget will be assessed. To do this we (/an independent agent) will need what can be described as “an auditable trail of intentions”, from identifiable policy objectives to identifiable programmes, with identifiable budgets, to identifiable outputs and maybe even identifiable outcomes.

There is an apparent complication. This auditable trail will not be a simple linear trail, because a single policy objective can be addressed by multiple programmes, and a single programme can address more than one policy objectives. Similarly with the relationship between a ministry’s programmes and outcomes in poor peoples lives. However, an audit trail can be mapped using a series of linked matrices (each of which can capture a network of relationships). These could include the following: PRS Policy Objectives X Ministry’s Programmes matrix, Ministry's Budget lines X Ministry’s Programmes matrix, and Ministry’s Programmes X Outputs matrix, and an Outputs X Outcomes matrix. This seems complex, but so is the underlying reality. As Groucho Marx said when his friend complained that life is difficult, “Compared to what?”

Postscript: Parallel universes do exist: Proof - a five year national plan with lists of policy objectives in the front and lists of programs in the back (with their budgets) but no visible connections between the policy objectives and programs & budget.

Identifying the impact of evaluations: Follow the money?

2005-04-02T10:01:00.000Z

Some years ago I was involved in helping the staff of a large south Asian NGO to plan a three-yearly impact assessment study. It was almost wholly survey based. This time around myself and a colleague managed to persuade the unit responsible for the impact assessment study to take a hypothesis-led approach, rather than simply trawl for evidence of impact by asking as many questions as possible about everything that might be relevant. The latter is often the default approach to impact assessment and usually results in very large reports being produced well after their deadlines.

With some encouragement the unit managed to generate a number of hypotheses in the form of if X Input is provided by our NGO and Y Conditions prevail then Z Outcomes will occur (aka Independent variable + Mediating variable = Dependent variable). Ostensibly they were constructed after consultations with line management staff, to get their interest and ownership in what was being researched. The quality of the hypotheses that were generated was not that great, but things went ahead. Questions were designed that would gather data about X, Y and Z, and cross-tabulation tables were constructed that would enable analysis of the results, showing with/without comparisons. The survey went ahead, the data was collected and analysed, and the report written up. The analytic content of the report was pretty slim, and not very well embedded in past research done by the NGO. But it was completed and submitted to management, and to donors. My inputs had ended during the drafting stage. The study then seemed to sink without trace, as so often happens.

A year or so later a report was produced on the M&E capacity building consultancy that I had been part of when all this had happened. In that report was a reference, amongst other things, to the impact assessment study. It said “The study also produced some controversial findings in relation to training, as it suggested that training was a less important variable in determining the performance of groups than had previously been thought. This finding was disputed at the time, but when [the NGO] had to make severe budget cuts in 2002-3 following the blocking of donor funds by the [government], training was severely cut. There is though still an urgent need for [the NGO] to undertake a specific study to review the relative effectiveness of different types of training.”

Learning circles and loops: Time for some more sophisticated representations

2004-12-22T09:58:00.000Z

I cannot count the number of times I have seen papers and books about M&E include a circle which shows an idealised learning cycle. One where planning leads to implementation which then leads to a review which then feeds back into planning. You can see dozens of examples if you do a Google Image search for "learning cycle". Look here for the Google search results

It is about time we developed some less simplified and more realistic representations of learning cycles. We can do this by situating our thinking about learning in more actor-oriented models of what is happening, versus models that focus on abstract and disembodied processes. And by thinking about multiple actors rather than single individuals going through their own action-reflection cycles.

The diagram below shows four technical units proposed for a new project management office in a national poverty program that I have been working with recently. Similar units could probably be found in many organisations. The lines show some expected information flows, especially in relation to the M&E unit. Reading the diagram, it is clear that the M&E unit is involved in a number of learning cycles, each of which involves a different actor (or unit in this case). Those learning relationships with other units will have different priority, which may well vary overtime. And the other units will be engaged in learning cycles with other units. It would be much more complex diagram if those other relationships were also included

The state of these learning loops can be investigated empirically, and documented by in an actor x actor matrix. [the actors can be individuals or groups of individuals, as found in each of the units] Cells in the matrix can detail what information has been received from the row actor by the column actor. Most Significant Change monitoring can provide the qualitative details of the information being exchanged between the two actors connected by a given cell. Then the relative importance of the information described in each row can be ranked by the provider (and the column contents ranked by the recipient). We can then examine how consistent the providers and recipients ranking are with each other. We can also compare the participants views with any formal model the organisation might have about how, what and when information should be being exchanged between the actors involved.

If you want to explore this approach further in your own organisation, and want to exchange ideas on how to do so, and the results, let me know Email the Editor

Open Government and Open Aid

2004-08-04T09:55:00.000Z

I have spent the last two weeks working with a government department in an African country that is charged with the responsibility of monitoring the progress with the country's Poverty Reduction Strategy (PRS). The PRS is meant to be a key government policy statement, which then becomes the focus of international donor support to poverty reduction efforts in that country.

Over the last few years donors have been supporting this department by funding the costs of producing annual and other progress reports on the government's implementation of the PRS. In the process of doing so they have obscured a useful signal of government commitment to the PRS: Are they will to invest their own scarce resources into monitoring the PRS? (This is a functioning government with an estimated GDP growth rate of 4.8% in 2003).

Another interesting signal is the government's willingness to publicise the PRS, and more importantly the progress reports on its implementation. There is a communications strategy, but it is not making much progress because of lack of resources (though they have been offered). Some reports have been printed and distributed but knowledge of their availability remains limited. They are not referred to on the departments website. In fact the department's existing website is notable for its invisibility. It is not linked to the government's main website, nor can it be found via search engine inquiries (But it is now being re-structured). Other reports have been produced and distributed within government, but because they have remained in draft status they have not been made publicly available. The focus has been on production of reports as mentioned in government agreements with donors, and much less on "dissemination" of those same reports.

Meanwhile donor support to the department is still coming through multiple individual donors, for multiple separate and overlapping activities. This does not encourage the development and implementation of a coherent and comprehensive plan by the department. Quite the reverse. There are clearly some perverse incentives at work. The department itself has been openly reluctant to share information about what others donors are doing, to individual donors, possibly because it gives the department more room to manoeuvre amongst the various donor agenda's and some freedom to pursue its own priorities (which remain out of sight). And individual donors are still being tempted to "cherry pick" specific activities for funding that could give them some influence on the PRS processes.

My current view is that this process needs to be radically changed. Instead of talking about "dissemination" of information about PRS plans, progress and revisions, the focus should be on developing and implementing a "disclosure" policy. Dissemination is a weasel word, a word that fails to say exactly what is meant. Dribbling out information, even within the government, could legitimately count as dissemination. But is that what is needed? No. A disclosure policy is different. It is a statement about what types of documents will automatically be made publicly available to anyone, without constraint. Printing 5,000 copies of a report may count as dissemination, but it does not count as disclosure. If those copies are available on a website, or information about their availability is made available via a website or public notice board or newspaper, then this does count as disclosure. What is crucial here is the extent to which there is handing over of control over access to information. Then anyone can theoretically participate in its use. This is quite different to various engineered "peoples participation" exercises being promoted by some donors as part of the PRS dissemination and revision process. These are by necessity limited in their scale and frequency, and much more expensive.

Disclosure policies need to be adopted by the supporting donors as well as the assisted department. They should commit donors to public disclosure to at least the following types of information: What government plans have they funded, what sort of support have the provided (including budgets) and what progress and financial reports have they received back? Ideally this information would be publicly accessible on the website of the government department being supported, where it can be seen in context. Not just on the donor's website. This is clearly ambitious. Right now the most immediate step that needs to be taken is to get the relevant donors to share this type of information amongst themselves, let alone with the public at large.

Some good examples of best practice need to be identified and promoted. One is the Government of Uganda website at http://www.finance.go.ug/peap_revision/ Here the government of Uganda has provided access not only to PRS documents , but also to various draft forms of these documents. Along with plans and progress with the PRS revision process, along with contact information about the key people leading the process. Unfortunately there is no link to and from the government's main website at http://www.government.go.ug/

Postscript: I have just noticed that PANOS are offering an award for writing on the subject of "Transparency, good governance and democracy:Do ICTs increase accountability?" Four awards of $1,000 each will be made for the best journalism on this topic produced by journalists in developing and transition countries.

No more paradigm changes please!

2004-08-03T12:40:00.000Z

"It is often assumed that participatory methods are suitable for
gathering qualitative information but that when hard, reliable,
numerical data are required we must turn instead to surveys and
questionnaires with their pre-determined categories and neat tick
boxes. In fact this is a myth, albeit one sustained by some with
vested interests in maintaining their "expert" status and privileges."

This is the first paragraph of a paper titled "Party Numbers: quantification though participation" which was published in the May 2004 issue of the Enterprise Impact NEWS letter [Issue 30]. This two page paper was a summary of a longer paper by both authors titled “Reversing the Paradigm: Quantification and Participatory Methods".

I have provided a brief critique of the two page summary, which is now available on the MandE NEWS website, here at www.mande.co.uk/docs/CommentsChambers&Mayoux.doc.

Amongst other things, my comments covers the following:
- the need for fewer loose references to paradigm changes
- less use of straw man arguments about different methods of impact assessment
- the need to think about which methods are appropriate in which contexts, rather than making broad generalisations about suitability of methods
- making more use of ranking methods, which are very simple forms of measurement that can be used in both inductive and deductive approaches to impact assessment
- limiting our ambitions about empowering people when doing impact assessments

After I emailed these comments to Linda and Robert, Linda then replied with her comments on what I had said. I have since replied to Linda with some further comments on issues she has raised.

Please add you own comments to this ongoing dialogue, by clicking on the orange "comments" link below.

Where have all the evaluations gone?

2004-07-12T12:15:00.000Z

Try finding the DFID Evaluation department on the DFID website, at http://www.dfid.gov.uk/ Not an easy task. Compare it to the SIDA website at http://www.sida.se/Sida/jsp/polopoly.jsp?d=107 ,where it very visibly placed on the first page

Try finding copies of recent DFID evaluations. The most recently numbered report is number 637, but there are only 125 actually listed on the website. There are about 10 per year listed from 1989 to 1998, but only 1 to 2 listed each year from 1999 to 2001, and none for 2003 onwards. Notably absent is the Development Effectiveness Report, produced in 2001 (though accessible via a key word search - but only if you already knew its title) And these are all listed in a section that is accessed through the Freedom of Information link on the DFID website!

Over the on the USAID EvalWeb website at http://www.dec.org/partners/evalweb/ there is evidence of a similar decline in the availability of evaluations. The website notes that "An impetus for this site is the decline in evaluations and assessments that are submitted to USAID's document repository, the Development Experience Clearinghouse. From 529 in 1994 to a projected 135 in 2003

Why the decline in evaluation activity, or at least the public availability of those reports, or at least information about their public availability? The increased channelling of DFID funds through sector wide and direct budget support funding mechanisms seems an unlikely explanation. Given the risks, especially with the latter, this development might be expected to generate more, rather than less, evaluation activity. And why all the gaps in the series of evaluation reports that are listed on the website? Why not list all that were undertaken, and if any are out of print, then say so.

Projects versus Project Funding Mechanisms

2004-06-30T15:22:00.000Z

Over the last few years I have had some involvement with the M&E of three project-funding mechanisms. One in the UK, one in Australia and one in south Asia. In all three cases almost all the thinking about the assessment of performance was focused on the analysis of the individual funded projects, along with some syntheses studies that were designed to make some more aggregate assessment of the results of particular categories of projects. The amount of attention given to the assessment of the project funding mechanism varied from a modest amount to none at all. I think this is almost the reverse of what should be the case.

All funding mechanisms that involve calls for proposals and then use a screening process to assess those proposals have in effect a theory of what makes a viable project. In as much as the people reviewing proposals feel they can rate some proposals as better than others then they probably also have a theory of what makes a good project, and a not so good project. These theories will be in the form of a view of what bundle of attributes, discussed during the review process, make the most difference to how successful a project is, in the short and long term. In an ideal world feedback from project-level monitoring and evaluation activities would lead to refinement of these theories about good projects, and this would be evident in changed selection criteria for accepting and funding project proposals. The funding mechanism would get better and better at spotting and funding good projects. In reality I have never seen this sort of feedback link in operation. At least in explicit form.

There are some broad types of theories that would be well worth testing, because they have some identifiable and significant consequences. One is supported by some prior experience. That is, it has been found that it is not the details of the proposed project activities, but the nature of the implementing partner that is what makes a difference between good and bad, or mediocre project outcomes. If this is true it could prompt a substantial re-weighting of emphasis in many project selection procedures, away from a focus on project activities and towards assessment of the project holding organisation. Another possibility is that there is in fact no significant correlation between how well proposals fit selection criteria and their subsequent performance. One possible response to these findings would be to slim down the project selection procedure and to intensify project monitoring and ongoing capacity building of funded projects.

Treating organisations as though they were machines

2004-06-20T17:02:00.000Z

The following comments are an excerpt from a response I made to the following paper by Alison Scott(DFID)"Assessing and Monitoring Multilateral Effectiveness", available online here

===============

The multilateral organisation as a machine

19. I was disconcerted to read section 9 on the use of multilateral's own assessment of their effectiveness. Not only about the multilaterals own lack of capacity to assess their effectiveness, but also the conclusion in para 9.3 that these efforts could not be used, and instead DFID would make its own judgements.

20. When we assess the performance of a machine we ask what is it doing and how does that match against what we expect it to be doing. When we assess the performance of an individual or an organisation, we also ask "what does s/he think they are doing" A person is expected to have agency; to be aware of choices and to make responsible choices. It is that awareness and responsibility which is the foundation of legal judgements that can make the difference between a death sentence, imprisonment, or freedom. On a more mundane level, it is an individual's (or organisation's) knowledge about what has happened which makes the difference between whether what has been done can be changed, avoided in future or replicated. The implication for MEFF is that DFID should be assessing the multilateral's knowledge about what it has been doing, and the effects of what it has been doing. That is what matters.

21. Fortunately most organisations know more than the sum total of what has been captured by their M&E systems. Knowledge is also captured in other documents, produced by other sections of the organisation. But more importantly it exists, often in tacit and informal forms, in the heads of people who make decisions about where resources should be allocated.

22. If DFID wants to engage with its multilateral partners, then one means of doing so is by trying to explicate their judgements of their performance, both the criteria they are using, the reasons behind those criteria, and the evidence of achievement on those criteria. This can then be complemented by independent verification by DFID, in the areas of performance that are of the greatest concern. A similar approach was taken with the assessment of a SIDA funded poverty alleviation project in Vietnam (See "A Study Of Perceptions And Responses To Poverty Within The Vietnam-Sweden Mountain Rural Development Programme".
==============
For the full text of my comments on the DFID paper go to http://www.mande.co.uk/docs/MEFFcomments.doc

Is moving the goal posts a good thing?

2004-05-08T09:35:00.000Z

I have just been reviewing the changes made in the Logical Framework used in the DFID-funded PETRRA project (rice research with and for poor farmers). Since the project started in 1999 there have been quite a few changes, at least once a year, if not more often. I decided to go back and compare the contents of the first Logical Framework developed in 1999, with those of the most recent version, last changed in mid-2003.

Somewhat to my surprise I found that many of the changes that had been made meant that the current Logical Framework was now more demanding, in terms of the expectations it set, than it was in 1999. There were now three, rather than one Purpose statements (heresy in some quarters, but viewed approvingly by the last OPR). There are now six Outputs instead of five (Communications work was given much more importance as the project developed). Of the original five Outputs, three had clearly developed a more demanding set of indicators. The other two were neutral, if not a bit more demanding. And the total number of indicators for all the Outputs and Purposes had grown from 18 to 30, roughly counted. Now an average of 3.3 per Purpose or Output.

And the project, which is due to finish by August 2004, looks like it will score above average on the achievement of most of the Outputs and Purposes, when assessed by the OPR team in July.

All this leads me to speculate about to what extent we could read changes in the contents of Logical Frameworks, as indicators of achievement (or lack thereof), even before we look at the evidence on the ground, or elsewhere. This might be half true at least. Where the Logical Framework has been scaled down, to be less demanding, that might relect a movement from unreality to realism, or from realism to failure to achieve. But even that difference should be possible to identify by reading the original Logical Framework. Another concern is sampling. How many changes were made to the Logical Framework, and how evenly over a period of the project? Many, spread well over a long period, would suggest the project managers had some ability/right to make changes. Few might suggest the latter. But that could be investigated. And the more changes there are made, the less likely they might be seen as "random pertubations" versus real trends.

Anyway,...food for thought.

Where are the partners?

2004-04-26T02:56:00.000Z

About a month ago I took part in the annual staff conference of a small UK NGO. The focus of the first two days was on identifying the main development issues that NGO should be addressing for the next three years or so. This was part of a wider strategic planning process that was just beginning. During the meeting the CEO made a point of distinguishing the NGO from others by the degree to which its approach was led by the views of its southern partners. If that really was the case, then I think the NGO would have had a justifiable claim to radicalism, something it was well known for in the past. But how could we verify such a claim? As I listened to the ongoing discussion about a range of important global issues, including HIV/AIDS, globalisation, fundamentalism, etc I notice how little, if at all, I could hear of the partner’s views on these issues. That the partners were not physically present in the meeting was not my main worry. I felt the CEO, and other staff, were well aware of the need for appropriate engagement with their partners. What concerned me more was that the discussion about global development issues made no reference to which partner thought what about which issue and why? Assuming the partners did have views, and had been consulted about them in the past, why was there no evidence of that process impacting on how the staff were presenting the development issues in this meeting? I would have thought that citing their partner’s views would have given extra weight to the views being cited. Associated with this concern of mine was a related feeling of far too much ungrounded analysis, that would then be very difficult to convert into a strategy that could be operationalised by the NGO.

The radical alternative would be to focus on the their partner’s views, and to talk explicitly about the areas of agreement and disagreement, both between their partners and with the UK NGO. This is where the NGO has some strategic choices to make (and choices it could fudge). Whose views should it support in future, how and why? And how should it respond to differences between its partners? Where should it seek new partnerships and why? Answers to these questions would help support claims they would like to make about working with, and even being led, by some of their partners. On the other hand, a continuation of talk about global issues without a focus on their partners views on those issues, would suggest they are trying to work through them, simply using them as means to an end.

Monitoring empowerment: A contradiction in terms?

2004-04-23T12:26:00.000Z

A colleague of mine has been doing some work for a major multilateral. They want him to help them identify some indicators of empowerment, which can be included in a national survey instrument. This has always struck me as a particularly paradoxical type of objective. The survey is trying to measure when someone else is empowered. But it will be the survey designer who will define what empowerment is. What if the respondents disagree that a particular development in their lives constitutes empowerment? Is this to be interpreted as "false consciousness" or is this actually an expression of empowerment itself (but probably unlikely to be recorded and analysed as a response)?

My advice to him was to treat diversity as an indication of empowerment. The rationale for this is spelled out in a conference paper I wrote in 2000, called "Does empowerment start at home? And if so how will we recognise it?". So for any given question about the attitudes or behaviour of the respondents, the survey analyst should examine the range of responses that were given (the SD to be more specific). Not the average response. Here is a quote from that paper:

At the population level, diversity of behaviour can be seen as a gross indicator of agency (of the ability to make choices), relative to homogenous behaviour by the same set of people. Diversity of behaviour suggests there is a range of possibilities that individuals can pursue. At the other extreme is standardisation of behaviour, which we often associate with limited choice. The most notable example being perhaps that of an army. An army is a highly organised structure where individuality is not encouraged, and where standardised and predictable behaviour is very important.

There was an associated footnote, which read:

As noted by some workshop participants, diversity in the behaviour of a set of individuals does not necessarily mean that all have equal choice. Inequalities of power (defined as choice) may still exist. Where we do find diversity in the set as a whole we could then do a more-micro-level analysis and examine the amount of diversity in the behaviour of one individual compared to another.

So, going back to the survey instrument being designed by the multilateral. As well as examining the range of responses to a given question, the researcher should also compare questions in terms of the range of responses to those questions Where is the most and least diversity of responses? Attention might then focus in on the questions with the least range of response. That is where further investigation would be potentially useful, to identify the nature of any common constraints limiting the choices people are making. And if anything can be done to address those common constraints.

What if all the respondents were sending their children to school, does that mean they are not empowered, within this diversity definition of empowerment? That could actually be the case, if there are legal sanctions against not sending children to school. It might also be true that most parents have little real choice about whether to send their children to school. In many developed economies parents are well aware that there are few livelihood options for adults without formal education. This apparently contrary example has some value. Not all forms of lack of empowerment will be of concern to those researching empowerment

Why did the chicken cross the road?

2004-04-20T16:19:00.000Z

On the Euforic website Rob van den Berg considers the challenges facing partners in evaluation....
"Collaborative evaluation is a potential minefield of misunderstandings about definitions, methodologies, concepts, logic and rationalities, reminiscent of the question ‘why did the chicken cross the road?’ The simple answer is that it wanted to get to the other side. Evaluation, however, wants to know whether the chicken took the shortest path across the road; whether it crossed the road in the quickest possible way; whether it did in actual fact reach the other side and whether it expects to remain there; and whether the needs of the chicken have been met by crossing the road..."

I think the case of the chicken who crossed the road has a lot of potential mileage as a metaphor for communicating what people think M&E is all about.

For example, my answer would be:

1. We need to ask the chicken what it had hoped to achieve by crossing the road. Not just pile on the questions regardless of its intentions. What were its objectives or its expectations? Did it in fact have a hypotheses it was going to test? A theory-of-change no less?

2. But we also need to be aware of the possibility that the chicken may have come across some unexpected benefits of crossing the road, after it did so. So just asking about its expectations will not be enough. We also need to ask the chicken about unexpected changes that took place. For example by using the Most Signficant Changes method, we could ask:

"What was the most significant change that took place in your life after you crossed the road?

We need to combine a deductive and theory based approach, with an inductive and experience based approach.

If you think the chicken would disagree with this, or you think there were other stakeholders in the chicken's neghbourhood who would have a different view, let me know, via the MandE NEWS Open Forum

Thinking about networks of policies

2004-04-12T12:22:00.000Z

I have just returned from XXXX in west Africa, where I have been working on PRSP M&E. One of my continuing concerns while there was to get a handle on the complex context in which PRSP M&E activities are taking place. As in most countries the PRSP exists in a complex policy context, it does not stand on its own. It links into, or is expected to link into, a number of other policies and associated implementation processes.

I think the relationship between policies is an area that deserves some serious thinking about, in M&E terms. There are at least two types of relationships that need to be considered:
1. The overlap in objectives of different policy documents.
2. The connection between policies created by information flows between them, once they are implemented and monitored

Policies can overlap in their objectives, this is fairly clear. New policies are often expected to overlap with existing polices. For example, new and specific policies in particular ministries might be expected to help articulate relevant sections of a PRSP. We can measure this overlap in at least two ways: (a) By examining the overlaps in sets of indicators used for M&E of both policies. The PRSP M&E Plan in XXXX has a useful table showing how a number of policies overlap in this respect. (b) By getting owners of two policies to rank the relative importance of their own and the other's policy objectives, and see how their rankings compare. I have done this with a UK NGO, to assess the alignment of a country level strategy with project specific strategies within the same country.

Policies can also be linked by information flows between them, once their implementation begins. In the best case, policy documents are integral parts of high level management cycles. They are plans, which are to be followed by implementation and then hopefully some sort of review processes. And then even more hopefully, some sort of revised policy. In other words, a higher-level version of the project management cycle (i.e. a policy management cycle). Where there are multiple policies, the M&E outputs of one policy management cycle can feed into the planning stages of another policy management cycle. For example, the Annual Progress Report (APR) of the PRSP (Poverty Reduction Strategy Paper) in country XXX is meant to inform the contents of the government's annual budget, as in many other countries. There is now some discussion about when is the best time for the APR to feed into the budget planning discussions, which take place in stages over a period of months.

In some cases there is a relative clear directional nature of these linkages. The APR is expected to influence the budget more than the budget influences the APR. In other cases the net direction of influence is less clear to me, at this stage at least. The World Bank's PRSC (Poverty Reduction Support Credit) includes indicators about the progress made with M&E of the PRSP. The APR should provide evidence of progress made with M&E of the PRSP, and help trigger flow of funds from the WB to the government. But the presence within the PRSC of specific indicators about PRSP M&E capacity may also shape how the PRSP is monitored. Whether it does or not, I have yet to find out.

There are of course many other policies that the PRSP might be expected to influence, and some of those may in turn be expected to influences the PRSP. Somehow these need to be identified and the desired linkages identified. Then we need to know enough about the stages of their respective policy management cycles to identify how and whether the linkages do actually work or not. Without this all the government's efforts put into "communicating the results" of the PRSP begin to look like a shotgun blast into the sky.

Right now I feel we have a very partial and incomplete view of how government and donor policies are and should be interlinking, through exchanges of information between their M&E stages and planning stages. It is the scale that is daunting, including the long cycle times that make them difficult to see as whole processes. The longer the cycle time of any policy management process the less likely that it will function in the same in ways as it was before. Right now we don’t know how the upcoming PRSP review and revision process will look like.

Question: How do you assess a country’s ownership of a PRSP?

2004-04-06T22:08:00.000Z

Answer: Bit by bit. There have been plenty of questions raised about the extent to which PRSP’s are really owned by the government of the country they refer to. (See Google search on ownership of PRSPs). But how do you assess whether a PRSP has country ownership? Well, maybe the way the question is asked could make a difference. One way is to ask who owns what parts of a PRSP. Rather than asking whether the whole document is owned by the whole government. In XXXX there are some PRSP objectives and associated indicators that could easily be adopted and owned by specific sections of government. For example, those relating to education or health, or macro-economic management. Okay, then how would you recognise when sections of government had taken ownership of specific objectives like these? Beyond simply saying so, which may not mean too much, these sections of government might actually collect and make information available about the associated progress indicators. Even stronger ownership might be associated with a detailed analysis of that data, as well as its collection and dissemination. In other words, the section of government would be investing its resources into M&E of their objective, and actually paying a cost in order to enable achievement of that objective. Back in country XXXX, the recently produced Annual Progress Report does not show any signs of any sections of government visibly owning specific sections of the PRSP. Nor is it clear who has been able to provide what information relating to PRSP indicators. In fact there has been an apparent unwillingness to explicitly state what information has not been made available by whom. The scale of lack of ownership has effectively been withheld from view.

Hypothesis-led Surveys of Influence - on KAP

2004-04-04T12:54:00.000Z

Today, on my day off, I have read through proposals received from two companies in Bangladesh to do an opinion survey of about 100 people in 20 organisations. The TORs (which I developed) ask a survey company to undertake a two stage survey process:

1. Interview PETRRA (a project) staff about which organisations they think their project has influenced most and least, and in what ways

2. Interview those companies to test out whether PETRRA staff hypotheses about expected influence are supported or not. Using open and closed questions and any other appropriate methods

A hypothesis led impact survey should produced a much more focused bit of evaluation research. The impact of the survey itself should be visible. If it finds the project staff hypotheses not supported by its findings then this should lead to changes, either in how PETRRA understands its influence, or in how the company does such surveys in future.

So far its not looking good. Both proposals have made no mention of the first round of interviews. They are just going to develop some questions of their own and then go out and interview people, and then presumably try to make sense of the results, without the bother of any guiding hypotheses! Oh well... it looks like we will have to provide them with a second round of briefing instructions and hope they get it right this time

Rick

PRSP Monitoring: Target fixation and mission creep

2004-04-03T08:40:00.000Z

Hi to new and returning visitors to MandE NEWS - from Rick Davies, Editor, MandE NEWS

This is a new step forward by Mande NEWS. I hope by starting up this Blog I might be able to generate some more content for Mande NEWS, on a more continuous basis. This will probably be more ad hoc and more from the hip, so some it will probably end up being deleted, later on in the cold light of day. Anyhow, here goes

Right now I am in XXXX, YYYY, working on monitoring and evaluation of the country's PRS (Poverty Reduction Strategy) Look here for Google findings on PRSPs http://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&q=prsp+poverty+reduction+strategy+paper&btnG=Search

When milestones become millstones: The Annual Progress Report (APR) on the implementation of the PRS is due shortly. The relevant government department is working hard to get it out on time. In the process the end purposes of such an APR are being lost sight of. Getting content is the main concern. Readibility will be a secondary concern, if there is time. Identifying the impact of the APR? Well, there has not yet been time to look at what happened with the last APR yet.

Mission creep at multiple levels in all directions: Donor and other comments are now coming in on the earliest draft of the APR. Could you explain x a bit more...? Why do you have no information on y...?

And this is response to an APR that is already try to track progress relative to indicators not just on the original PRS but at least four other policy documents that have come into the picture since the PRS was written. These include:
- a summary revision created by the government when it came into power
- the Poverty Reduction Support Credit, a WB device
- Multi-Donor Budget Support policy document
- HIPC triggers
- Milleniuum Development Goals (okay, they were there before the PRS)

Needed?: Some continual and public mapping of how the various poverty related (govenment and donor) policies relate to each other (or not), in terms of overlapping indicators and objectives. Both existing and planned policies.

Postcript: 6 hours later, my laptop hard drive leaves this world. The second in 18 months. I will not be buying another HP laptop! Fortunately I have been backing up reasonably often, and I am carrying two memory sticks (much recommended)