There are six broad ways for gathering information. Whatever approach you use, the actual methods will fall into one of these categories.
• Survey – questionnaires, checklists
• Interview – structured or unstructured, getting first-hand responses from end users
• Desk Study – reviewing existing documentation: reports, publications, web sites
• Observation – visiting the project site and personally observing what is happening
• Focus Group Discussion – facilitated meetings with groups of end users around a particular issue
• Case Study – an in-depth investigation over time into one particular end user’s experience and outcomes of the programme
Each method has:
• An overall purpose
• Limitations and Risks
The survey method is appropriate when we need to get a lot of information quickly. Relatively easy to design and administer, questionnaires / surveys have benefits of being:
• Non-threatening: there is no-one to judge responses, so end users do not have to ‘please’ the questioner
• Anonymous: end users can freely respond knowing that their identity will be kept private
• Relatively cost-free
• Easy to compare: makes analysis easier (if the questionnaire is designed with the analysis in mind!)
Things to consider are:
• Respondents might not complete carefully
• Questions have to be designed carefully in order not to bias the response from the end user
• This method does not develop any kind of relationship with the end user
• Surveys never get the ‘full story’
Interviews help us gain some more depth and insight – to really understand how the project is affecting end users or how they feel about it. Here we get both range of information – we can cover a lot of issues – as well as depth. We can respond immediately and investigate deeper any responses, while developing a relationship with the respondent.
However, interviews take time. While the interview may be structured (i.e. the same questions are asked to all respondents), their flexible nature means that analysis and cross-comparison may be difficult: after all, everyone has a different story. There is also the danger that the questioner may bias the responses, or that the respondent may give face-saving answers.
3. Desk Study
Reviewing secondary documentation is a good way to find out how a project is doing without actually interrupting the process. This is from a review of reports and other project documents.
In the best scenario, we have access to a huge amount of information without having to interrupt the programme’s operations. However, it can be time consuming; we need to know what we are looking for or can get lost. Sometimes the information is incomplete or cannot be verified. It’s also inflexible: you can only use what already exists.
Observation – for example, through visits to project sites – lets us get verifiable, first-hand information about how things are actually done. It is very useful for observing processes as they happen – for example, observing a User Group Meeting.
Despite being time consuming, a big advantage is that we can respond immediately. We can ask questions to gain further understanding of what we see, and instead of asking ‘what if?’ can actually take action and see ‘what happens’.
However, understanding what we see isn’t always easy to interpret, and categorising the information collected can be tricky. Another drawback of observation is that our presence can influence others’ behaviour, and what we see may not be typical of what happens when we are not present.
5. Focus Group Discussion
Originally developed as a quality tool in marketing, the Focus Group Discussion (FGD) is useful for exploring group perceptions on a topic in depth. These could be reactions and feelings of end users or exploring group concerns. FGDs an also help us to resolve emerging conflicts and reach participatory decisions.
The FGD is efficient in that we can get both range and depth of information in a short time. It also serves to communicate key information about our programmes to end users.
• The limitations of this method include:
• Scheduling the discussion can be can be difficult
• Analysing the discussion afterwards may be complex
• We need to ensure that the FGD is conducted by a skilled facilitator
• Participants may not give honest responses, and try to please the facilitator or fit in with the group
• Relationships between the group members may mean that some members do not express their ideas openly
• The culture of the group may not lend itself easily to the divergence of opinion necessary to reach true consensus
6. Case Study
The Case Study focuses on depth, and aims to fully understand a particular end user’s experience of a programme. It gives a full picture of the end user’s experience of the project inputs, processes and results, providing a powerful way to demonstrate the benefits of the programme to outsiders – look at how many INGOs use case studies in their fundraising with the general public.
The limitations are that:
• It is very time consuming to collect
• It has to be planned and conducted from the start of the programme – not just added later. (That’s a ‘Success Story’)
• The Case Study gives depth of information, not breadth
• Analysis and, particularly, cross-comparison, can be difficult