Research Design Training of Trainers: - PowerPoint Presentation

1. Research Design Training of Trainers: Module 2 MethodologyWebex, May 2020

2. Research Design Training of Trainers: Module 2.1 Methodology design (Research methods)Webex, May 2020

3. Session ContentsOverview of research methodsDistinction between quantitative & qualitative researchTypes & applicability of different research methodsQ&A

4. 1. Research method overview

5. Research methodologyThe methodology is an outline of the overall data collection and analysis strategy that will be used to implement the research cycleThe methodology should:Be compatible with the preliminary data analysis planBe designed in a way that ensures the intended scope of the research (i.e. objectives and research questions) can be feasibly achieved to the required quality, given the time, resources and access availableDesigning a methodology involves three key components: Selecting the overall research methodSelecting the appropriate data collection approach(es)Designing the sampling strategyOur focus for today! 

6. Categories of research methods Research methods are broadly distinguished between the following categories:QuantitativeMeasure prevalence of issues, verify hypotheses and establish causal relations between variablesLarge samples, structured data collection, and predominantly deductive analysisQualitativeExplore and discover themes, develop theories, rather than verify hypotheses and measure occurrences Smaller samples, semi-structured data collection, inductive analysisMixed MethodsCombines both qualitative and quantitative to (1) collect and analyse both types of data and (2) use both approaches in tandem

7. Deductive (quantitative) vs. inductive (qualitative) analysis approach

8. Selecting your research methodWhat factors to consider when choosing one research method over another?Overall applicability to meet research objectivesTime i.e. key planning and decision-making milestones to informResources availableMaterial resources Financial resourcesHuman resources Access to population of interest

9. 2. Quantitative vs. Qualitative research

10. Differences between quantitative & qualitative researchThe distinction between quantitative and qualitative research is not always as clear-cut: Individual and household surveys Commonly associated with quantitative, large sample research Could also be used for a qualitative case study Key Informant interviews and community discussionsCommonly associated with qualitative, semi-structured data collection & analysis Could also be used for quantitative data collection & analysis where time and resources do not allow a large, representative sampleFocus Group DiscussionsPerhaps the most distinctly qualitative research method, always administered using a semi-structured data collection tool Often analysed using content analysis i.e. a somewhat quantitative approach counting the number of times a theme or set of words appear with the discussion transcriptsThis content analysis can then inform the more in-depth qualitative analysis.

11. Differences between quantitative & qualitative researchDistinction between the two can be made based on the following three criteria: QuantitativeQualitative1. Type of data collectionStructured, close-ended data collection tools Semi-structured (but not unstructured) data collection tools2. Type of analysisMeasuring prevalence, quantifying issues, and primarily involves deductive analysisExploratory, and primarily involves inductive analysis3. Type of sampling strategyCan use both probability or non-probability sampling  generalisation to the wider population possible Non-probability sampling  generalisation to the wider population not possible

12. 3. Types & applicability of different research methods

13. Types of research methods (1)CategoryType of research methodsDescriptionWhen to use this methodQuantitativeStructured, probability sampling/ censusStructured, close-ended data collection; Quantitative analysis; Data collected from a census or through large samples, with sample size calculated based on probability theoryTo measure prevalence and make generalizable claims,To conduct deductive analysis (relationship tests, verify hypothesis)To identify key factors that influence a particular outcome or understand the best predictors of a specific outcomeQuantitativeStructured, non-probability samplingStructured, close-ended data collection;Quantitative analysis; Can be small or large sizes; non-probability samplingTo measure prevalence (indicative only) but contextual and/ or logistical constraints do not allow for large, repressentative samplesTo draw indicative inferences from a sample to a population

14. Types of research methods (2)CategoryType of research methodsDescriptionWhen to use this methodQualitative Semi-structured, non-probability samplingSemi-structured data collection;Qualitative analysis; Relatively small sample sizes; non-probability samplingNo measurement of prevalence or verification of hypothesis needed; No or limited prior understanding of the situation to be studied and the specific variables to be assessed; To conduct inductive analysis i.e. explore and develop a theory or pattern of meaning, based on experiences, observations and perspectives of the situation being studiedMixed MethodsN/ACombines both qualitative and quantitative methods, both in terms of collecting and analyzing both types of data but also using both in tandem to enhance the overall strength of the studyQuantitative or qualitative methods by themselves inadequate to understand the research problem; To use all methods possible to obtain an in-depth, comprehensive understanding of the research problem.

15. The most powerful research method? Mixed methods research – if time, access, resources allow! Common misnomer that quantitative research is the strongest – not always! Not all issues need to explained in a quantifiable waySome issues are over-simplified if only explored in numeric terms In-depth explanation and contextualisation is usefulUltimately depends on the research objectives

16. Questions?

17. Research Design Training of Trainers: Module 2.2 Methodology design (Data collection approaches)Webex, May 2020

18. Session ContentsUnit of measurement Types of data collection approaches (structured)Types of data collection approaches (semi-structured)Types of data collection approaches (mixed methods)Frequently Asked Questions (FAQs)Overview of remote data collectionQ&ATask for the week

19. Unit of measurement

20. What is it?The unit that will be used to record, measure and analyse observations/ information collectedExamples?IndividualFamilyHouseholdCommunity/ groupTown/ villageFacilityCow

21. Remember… Unit will impact the time, resources needed to collect and analyse informationUnit will define the depth of information possible and scope of analysisDepth of informationLocation levelHousehold levelIndividual levelCommunity/Group levelTime / Cost / AccessIt is possible to aggregate from a lower unit of measurement upward (e.g. household to community) but rarely vice-versa

22. Data collection approaches: Structured

23. 1. The structured survey approachInformation collected through an interview, a discussion, a conversationUsing structured, close-ended data collection tools Collection of quantifiable informationCross-sectional or longitudinal Types of data collection methods? Household (HH) survey – collecting data at HH level, to understand experiences and characteristics of HHs within population of interest Individual survey – collecting data at individual level, to help understand situation and characteristics of individuals within the population of interest  can include some HH level indicators if neededKey informant interview – collecting data at community. location or group level from a key informant (KIs) i.e. an individual whose informal/ formal position gives him specific knowledge about other people, processes, or events that is more extensive, detailed, or privileged than other individuals in their group/ community/ locationGroup discussion – collecting data at community, location or group level from a group of representatives e.g. KIs

24. 1. The structured survey approach- ApplicabilityWhen should you use this approach?To measure prevalence  provide a quantifiable, numeric description of the trends, behaviours, experiences, attitudes or opinions of a populationTo generalize findings to a wider population  probability sample  statistically representative informationNeed prevalence data, understanding of scale of crisis but probability sampling not possible  non-probability sample  indicative information Types of research cycles this approach is commonly used for? Multi-sector needs assessments In-depth thematic needs assessments e.g. WASH Cluster needs assessment Longitudinal studies Third party monitoring (impact evaluation, outcome monitoring, post-distribution monitoring, etc.)

25. 2. The structured experimental approachWhat is it?Similar to survey approachBut relies on experimental survey design  control vs. treatment groupTypes of data collection methods?Household (HH) survey – collecting data at HH level, to understand experiences and characteristics of HHs within population of interest Individual survey – collecting data at individual level, to help understand situation and characteristics of individuals within the population of interest  can include some HH level indicators if needed

26. 2. The structured experimental approach - ApplicabilityWhen should you use this approach?To measure prevalence and evaluate the outcomes or impact of a medium to large-scale intervention on the population of interestGeneralize findings to a wider population  probability sample  statistically representative informationTypes of research cycles this approach is commonly used for? Outcome monitoring Impact evaluationsEtc.

27. 3. The structured observation approach (Description)What is it?Information collected through observation rather than conversationUsing structured, close-ended checklists to collect quantifiable informationLooking for specific object, behaviour or event against a checklist e.g. Household using soap? Damage to health center? Students participating in classroom?Can be used as part of experimental approachTypes of data collection methods?Participant observation – researcher participates in context (e.g. anthropologists)Direct observation – researchers observes context (e.g. psychologists or clinical research)

28. 3. The structured observation approach - ApplicabilityWhen should you use this approach?Serves similar purpose as survey approachDepends on research objectives  observation vs. conversation? Types of research cycles this approach is commonly used for? Could be same as survey approachCould be same as experimental approach

29. Data collection approaches: Semi-structured

30. 4. The semi-structured discussion approachWhat is it?Information collected through detailed, narrative interviews, group discussionsUsing semi-structured (NOT UNSTRUCTURED) data collection tools  open-ended questions, probes Purposefully selected participantsTypes of data collection methods? Individual interview – collecting data at individual level, to help understand situation and characteristics of individuals within the population of interest  can include some HH level indicators if neededKey informant interview – collecting data at community. location or group level from a key informant (KIs) i.e. an individual whose informal/ formal position gives him specific knowledge about other people, processes, or events that is more extensive, detailed, or privileged than other individuals in their group/ community/ locationGroup discussion – collecting data at community, location or group level from a group of representatives e.g. KisFocus group discussion – bringing together people from similar backgrounds or experiences to discuss a specific topic of interest; data collected at community, location or group level

31. 4. The semi-structured discussion approach - ApplicabilityWhen should you use this approach?To gather detailed insights about the experiences, perspectives of specific population group or locationTo provide a qualitative description of the experiences, trends, attitudes or opinions of a populationTypes of research cycles this approach is commonly used for? In-depth assessments where there is limited prior understanding of a situation e.g. access to cash among refugees & migrants in Libya Participatory mapping exercises (mapping FGDs or KI interviews)

32. ‘Most Significant Change’ data collection techniqueA very specific type of participatory, discussion-based data collection method used for monitoring & evaluation Invites participants (through KI interviews, individual interviews or FGDs) to explain the most significant changes brought about in their lives by a project over a given period of time, in key domains of changeUseful for third party monitoring or impact evaluation research cycles

33. ‘Most Significant Change’ data collection techniqueThe stories, anecdotes you collect from beneficiaries/ project partners, broken down by «domain» of interestThe stories, anecdotes you select to qualitatively analyse change per «domain», in consultation with project team

34. 5. The semi-structured observation approachSimilar to structured observation approachBut two key differences: Structured observationSemi-structured observation1. Differences in data collection methodsInformation collected using a structured set of questions, usually to identify specific object, behaviour or event against a checklistInformation collected based on a short set of open-ended questions for observations e.g. movement patterns of refugees in and out of camps during a sustained period of time2. Differences in purposeProvide a quantifiable, numeric description of the trends, behaviours, experiences, etc. of a populationGather detailed insights about the behaviours, experiences of a specific population group or location, and to understand, by observation, how things are done and what issues exist

35. Data collection approaches: Mixed Methods

36. 6. Sequential mixed methods data collection Method used to sequentially elaborate or expand on the findings of one type of research method with another1. Exploratory sequential approach2. Explanatory sequential approach3. The “ideal” sequential approach

37. 7. Concurrent mixed methods approachMethod used to merge or converge the findings from different research methods collected at the same timeAlternative to sequential approach if time constraints  sequential better practice if time and resources allow Concurrent mixed methods serves two key purposes: Triangulation strategyEmbedded strategy

38. Case study data collection techniqueUsing a combination of different data collection methods to zoom in to a specific issue, area or groupA component within a research cycle, not a research cycle by itselfUseful to collect detailed information on an event, activity, process, group e.g. zoom in to one specific type of intervention in an area within a larger DFID-funded humanitarian programme

39. Frequently Asked Questions (FAQs)

40. FAQs (1)What is the difference between a key informant interview and an individual interview? Isn’t the key informant also technically an individual?The differences lies in the unit of measurement  individual experiences (individual interview) vs. community/ village/ institution experiences (KI interview)For semi-structured data collection, when is it recommended to use FGDs over KI or individual interviews?This depends on two thingsResearch objectives and type of information needed e.g. Variety of opinions and experiences useful? Specific information needed from an expert? Topics sensitive to discuss in group setting?Logistical constraints e.g. Large number of individuals to be reached within a short timeframe?

41. FAQs (2)Is it possible to have two different units of measurement in the same questionnaire?Ideally, should be avoided, but there are some exceptions: Individual information within a household survey (e.g. child attendance roster)Household information within an individual survey (e.g. household size or income indicators)Individual information within a village/ community/ location level interview (e.g. KI’s displacement status and experiences, if KI also part of the affected population)Household information within a village/ community/ location level interview (e.g. KI estimates # or % of households affected by a specific situation in a village)What if my population of interest includes minors (i.e. individuals <18 years of age)? Can I collect data from minors?Only if absolutely necessary to meet objectives of the researchOnly if required information cannot be collected from adult respondents e.g. parents or caregiversIdeally, only from respondents >15 years Only if the required protocols are being followed Will de discussed later in this training 

42. Questions?

43. What methods to use if you don’t have access to the population of interest?

44. What is remote data collection?Remote data collection is a means of gathering data without a physical presence in the data collection location and without direct, in-person contact with the population of interest When is it useful? When it is not possible to conduct in-person visits to the locations / populations of interest because of reasons such as:Disease outbreak (e.g. COVID-19) Time or resource constraints (e.g. not enough budget to hire enumerators to cover all areas for face to face interviews)Access constraints due to: Security concernsCOVID-19 travel restrictionsPhysical access barriers such as lack of infrastructureSevere weather conditions which limits travel possibilities, etc.Etc.

45. Pros and cons of remote data collectionProsConsPlanning efficiencyMore time and resource efficient; if necessary logistics already in place, could be fairly straightforward to deployChallenging and time consuming to set up correctly (e.g. identifying respondents, organizing necessary logistics, etc.), difficult to apply stratification in sampling; challenging to monitor progressImplementation efficiencyEasier to implement even with limited time, access and resources (assuming planning and design is done robustly)Higher likelihood of low response rates; limited means of verifying responses/ data quality assurance; more challenging to build trust with the respondents; difficult to deploy long or complicated questionnaires CoverageEnsures maximum possible coverage of areas and population of interest despite access constraints Difficult to have the “full picture” as it could introduce potential sampling biases (e.g. based on phone network coverage) and results in exclusions/ oversight of certain population groups or areas

46. Some types of remote data collection methods (1)Phone-based (individual, household, community level)Most relevant for: needs assessments, post distribution monitoring (PDMs), humanitarian situation monitoring (HSM)Representative sampling could be possible2. REACH “Area of Knowledge” methodology (face-to-face data collection in alternate location)Most relevant for: community-level needs assessments or HSMRepresentative sampling not relevant (requires identifying the respondent most likely to have the required knowledge)3. Internet-based data collectionTools include: social media, web-based surveys, online discussion platforms, chatbots (WFP mVAM), etc.Most relevant for: community-level needs assessments or HSM (KI interviews or group discussions), PDMs (individual perception surveys) Representative sampling could be possible (but extremely difficult to implement e.g. would need email address database and usually low response rates)

47. Some types of remote data collection methods (2)4. Remote sensingOnly relevant if aim is to gain an understanding based on specific physical characteristics of an area (e.g. agriculture and vegetation health analysis, shelter damage assessment, flood impact assessment, etc.)Representative sampling or even census could be possible5. Secondary data review and “expert” consultationsMost relevant for: needs analysis or HSMOnly feasible if relevant and «reliable» data sources already exist6. Paper form submissionsOnly applicable if respondents have no movement restrictions and are able to send paper forms back through required meansLogistically difficult, not the most time and resource efficientMost relevant for: community-level needs assessments or HSM (KI interviews), PDMs (individual perception surveys) Representative sampling could be possible (but extremely difficult to implement e.g. would need postal address database and expect very low response rates)

48. Post-distribution monitoring (PDM) of cash assistance and core relief items to refugees and IDPs across IraqProject began in 2016 and remains ongoingData collected through two call centres: Erbil and BaghdadHousehold level data collection, providing at least a 90% confidence level and 10% margin of error at Governorate levelPhone-based data collection example: Iraq UNHCR Cash Assistance PDM (2017-now)Project backgroundTo improve time and cost efficiency, since most of the data collected would not be verifiable by enumerators in the fieldAccess to beneficiary contact lists ensures time-efficient data collectionThe project has a wide geographical spread, so the call centre allows for rapid, far reaching data collection Why was it remote?What worked well?Challenges?A team of enumerators have been well trained and dedicated to this assessment continuouslyAvailability of anonymised, comprehensive beneficiary lists for sampling purposesRemote data collection helps ensure data privacyTypically the call centre remains functional, regardless of changing access constraintsBuilding trust among respondents Ensuring respondents understand the role of this assessment Potential for duplication as beneficiary lists were at the individual level while sampling was at the household levelSpace constraints within the call centre during multiple ongoing assessments

49. Humanitarian Situation Monitoring in ‘hard to reach areas’ of ‘3 border’ area between Mali, Niger and Burkina FasoSince November 2019Remote data collection through face to face interviews with KIs who travel between accessible and inaccessible areasCollect information about humanitarian situation in each country / areas with same tool to allow for comparabilityAoK data collection example: 3-border HSM in Sahel (December 2019- now)Project backgroundTo gather information about areas where humanitarian access is low or unreliableTo ensure supply of information about these areas is regular and not contingent on access, allowing for trends monitoringLess resource intensive – good compromise to gather indicative data in complement to existing, more robust data collection systemsWhy was it remote?Once knowledge of population movements within a region is clear, easy to set up data collection to ‘capture’ information about different areasAbility to cover data across a vast territory from a handful of static bases.Ability to monitor trends on situation in hard to reach areas and to compare and contrast between severity levels.What worked well?Reliability is not high and ability to verify validity of data collected is low – it’s indicative onlyKIs reporting on overall situation at settlement level can hide inequalitiesWhile it is less challenging finding KIs from relevant geographic areas, it can be difficult to find a balance of KI profiles (men, women, age groups, vulnerable groups etc), impacting comparative analysis.Challenges?

50. Now available: SOPs for Data Collection during COVID 

51. Questions?

52. Next session?

53. Task for the week

54. InstructionsTake the research objectives & preliminary analysis plan you formulated last week and briefly determine:Which overall research method would be most appropriate and why?Which data collection approach(es) would be most appropriate and why?It is up to you to decide whether you want to assume face-to-face data collection is possible/ remote data collection is necessary in your scenario Don’t go into sampling just yet, we will come back to that next weekIs there likely to be any sensitive information collected? Is this suitable to the data collection approach being discussed?What additional information do you need to make final decisions on the approaches?We can discuss how this goes next week!