How can we get better at working with evaluation tool kits?
At Apticon last week Martin Belcher (Aptivate) proposed a discussion around evaluation tool kits. He introduced the session and wanted to talk about the design and use of evaluation tool kits. Asking how do we do it effectively? why are the so many out there? What are the tools we can use to evaluate intervention? He had a particular interest in the because of his involvement in a the World Bank project where we are creating a Guide to Evaluating Digital Citizen Engagement. We talked about how there are many options for evaluation frameworks out there but it's unclear which are good or bad and how we should choose one.
Alan Upstone (Metalab) explained how the ‘Development industry are trying to invent things that are already available in other industries’ and how there is an element of ‘Not invented here syndrome’.
We discussed whether there is a best practice way to structure and write a Logframe and could there be suggested ways to write your outcomes and activities. George Flatters (Aptivate) asked if there is a Github equivalent for this kind of thing? and Chris Wilson (Aptivate) replied ‘There almost is, i’m sure, we could make one’. This sparked discussion around how we can reuse what others have done and why this is hard. 'Ngo’s don't have funding to help other people use the tools they have created.'
Alan recalled a time where he said to colleges ‘McDonalds wouldn’t do it like this' and people looked at him horrified and said ‘we’re not doing McDonalds were changing peoples lives!’ we went on to discuss that often frameworks have been created with a particular project I’m mind and no one is creating a generic one that others can use. Then asking, if that was to happen, how would it be shared? and how would it be published? How do people feedback on their experience? and how are upgrades made?
Martin described that the essence of a lot of these evaluation frameworks as: Who did you engage with? where they the right people? and what were the results?
Chris talked about how it's hard to compare two projects with different frameworks and decide which was better, even if one seemed to provide better results it doesn't mean it would have been better for the other project.
Alan talked about how he finds this kind of thing hard to work with. ‘Log frames are hard to use, hard to know about the options. You look at the 30 you find and choose your top three but there is actually 40 and 10 you don't know about’ ‘Its very hard to do the research and find the right one, it can be easier to just choose one and get on with it.’
George said ‘It needs a powerful and influential name behind it, then it’s going to be more successful.’ and that ‘it sounds like a comms and marketing strategy is needed to go with it’
Liam Finnis (INASP) asked ‘Are you building this now’ and ‘Have you looked into the others?’. Alan said ‘I would like to see the list of options you’ve got. Have you made it available? Can I search on google and find it and add to it? I want to be part of the solution! Why are the ones you have seen not good enough?’
Martin replied that ‘In my personal opinion there is one that is good enough’ and that they have been taking inspiration from this. He went onto explain that there is an element of the World Bank wanting to create one that they can be behind. He went on to explain that he wants to avoid this work not being used and not being usable, asking ‘Can we build in features to make it easy and reusable, so that people can easily download it online for eaxample?’
Chris asked ‘has it been discussed that this is going to need an ongoing support & updating regularly?’ and this brought up questions like ‘Can we build up case studies of use and show examples/evidence of lots of people using it effectively?
We started talking about the visual design and user experience on the service, George pointed out how important it is that it looks good and is easy to use. Alan agreed saying that IATI ‘was just so painful to use’. He talked about how it should be free to use and said that if he said to people ‘Aptivate are building one and it’s going to be free to use, they would say, we'll use it’. We talked about how with the World Bank behind the project we can go to NGO’s and say why don’t you use this? and that people might like the idea of being early adopters.
Liz Fearon (London School of Hygiene & Tropical Medicine) said that ‘Accessibility should be part of your process. We often talk to NGOs and i’m on the academic side. I say to them ‘We need to be involved from the planning phase’ it’s too late otherwise’ Alan said that if he was working in a tool he would like to have someone like Liz to review the information he had put into the tool and evaluate his work ‘That would sell it to me’ he said.
Martin talked about World Bank's aims for the project explaining that ‘Not only do we want it to create a framework, we want it to be rigorous’. The session provided some good ideas and things to consider in achieving this. To keep up to date with our projects Sign up for our newsletter and to find out more about Apticon events see our Meetup group.
There were a few things referenced in the discussion:
- OECD Dack framework
- Tegemeo - Design research questions
- IATI
- Evaluation flash cards - tool kit with a good structure, short but replicate lots before it