Rory Dillon – Development Management MSc 2012/2013

 I have been wondering if our (the wider interested development community) remonstrations with UN agencies over how they could improve their projects have simply fallen on deaf ears. We have found these new approaches to measuring impact – randomised control trials or community participation – that would revolutionize the way they operated if only they would get involved. Are the UNDP, FAO and UNIDO (or for that matter Oxfam and ActionAid) simply smiling and nodding and trying to get on with ‘the job at hand’?

When I began my masters at LSE, I saw the problem of development as one of reallocation. There are people living on less than $1 a day and people living on significantly more and the problem is to rebalance the scales. In the course of my studies, I became more interested in how well charities and aid agencies use donations that they are given. Even if not enough people are receiving treatment, does the treatment work?

The short answer is, sometimes. The aid world is analogous to the world of medical science in the Victorian Era: old wives’ tales compete with proven science for the attention of those who have the power to spend money. To extend the analogy, aid practitioners require evidence to separate the two. As it came time to plan my dissertation the questions which came to prominence for me were: are aid agencies measuring the impact of their projects as efficiently as possible and if not, what factors stop them from doing so?

I particularly wanted to test hypotheses about constraints to improved impact assessment. Writers such as Bill Easterly and David Mosse have written about mechanisms which may give incentives for employees within UN agencies or charities to avoid rigorous monitoring. Was high quality measurement restricted by:

• Lack of knowledge: Were evaluators simply unaware of how to improve impact assessments?
• Over-claiming: Did project staff fudge reporting to project budgets and jobs? Was there, for example, pressure to sell positive stories to management or donors?
• Complexity: Were projects too complex to measure well? Projects which rely on many interrelated factors?
• Laziness: If projects were difficult to measure, how did that affect morale? Did it sap enthusiasm for efforts to improve monitoring?

To answer these questions I chose to make a case study of the Food and Agriculture Organisation, the UN agency with the mandate to improve global agricultural productivity with a budget of $2.4bn. In particular, I worked with the Fisheries Department and investigated a recent evaluation of the impact of projects associated with the FAO Code of Conduct for Responsible Fisheries. I spent a fortnight at FAO headquarters in Rome. I conducted an analysis of the Fisheries evaluation and conducted interviews with evaluators and project delivery staff.

My analysis of the Fisheries evaluation showed that it was high quality with important caveats, the most important being the lack of baseline data collection. The evaluation aimed to see how well Fisheries projects helped countries implement the Code, but since these projects had not measured Code implementation before the project began, a rigorous evaluation of the impact was stymied.

So, why didn’t staff collect good baseline data? Was it because evaluation staff did not understand it was necessary? No: in interviews, staff showed that they were well aware of best practice in conducting good impact assessments – spontaneously mentioning key elements of best practice. Did project delivery staff seek to actively bend or twist reported information?  No: evaluators were in consensus that ‘overclaiming’ was not a significant problem. Project complexity figured highly in project staff descriptions of the constraints to improved monitoring and interviewee comments showed that several were sceptical of the ability of managers and evaluators to measure those complex projects effectively.

Project staff had mixed incentives:  in theory, they supported the idea that performance measurement would ensure projects worked well but they also feared that, in the short term, baseline data collection would crowd out time used in delivery.

It is not enough to assume that charity and multilateral agencies will act in their own long term interests if their short term interests are in contradiction. Campaigns to improve measurement within large UN agencies and charities need to factor in the incentives of workers if they are to affect beneficiaries. Otherwise, development agencies will be left looking both ways on impact assessment, saying ‘yes’ but meaning ‘no’.

Rory completed his Development Management  MSc last year and is now working at Link Ethiopia, a charity committed to widening access to education in Ethiopia. He is also lecturing on the Business HND programme at St Patrick’s college.