/
Reviewing Surveys, Interviews, and Focus Groups Reviewing Surveys, Interviews, and Focus Groups

Reviewing Surveys, Interviews, and Focus Groups - PowerPoint Presentation

stefany-barnette
stefany-barnette . @stefany-barnette
Follow
342 views
Uploaded On 2019-12-10

Reviewing Surveys, Interviews, and Focus Groups - PPT Presentation

Reviewing Surveys Interviews and Focus Groups A PostEvent Review to Handling Surveys and Interviews in NVivo 10 Jan 30 2015 Why The PostEvent Review Survey and interview data are extracted from particular research designs and approachesfrom particular contexts ID: 769942

research data survey coding data research coding survey sampling nvivo questions time information group analysis based cont groups focus

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Reviewing Surveys, Interviews, and Focus..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Reviewing Surveys, Interviews, and Focus Groups A Post-Event Review to “Handling Surveys and Interviews in NVivo 10” Jan. 30, 2015

Why The Post-Event Review…? Survey and interview data are extracted from particular research designs and approaches…from particular contexts. The way questions are asked affect the data and the structure of that data The research approach affects the way coding of that data should be done The research approach affects what is knowable 2

Main Contents Research methods Generic information about…: SurveysInterviewsFocus Groups NVivo 10 demos Disclaimer : This slideshow is only a first rough-cut to set the stage. For most, this is a refresher. The main message is that various types of research procedures result in different types of information that may be analyzed in particular accepted ways, with limited particular assertions possible. The assertability of the data comes from theory and method, domain practices, and other elements, and not the data analytic tool (NVivo) per se. 3

(Generic) Research Design Conceptualization: Research Objectives, Research Questions, Hypotheses Review of the Literature (annotation and write-up) Research Design (methodology) Instrumentation (design and pilot-testing) Sampling (selection of respondents) Research Data Collection (sometimes multi-method, mixed method; form of data affects analysis) Follow-up (if needed) Data Visualization Data Analysis (quantitative and qualitative methods)Discussion Reporting Out Future Research 4

Surveys 5

Basic Purposes of Surveys Collect data about people’s experiences, situations, attitudes, beliefs, opinions, and other factors at a particular point-of-time, or over time Complement various other types of research, including experimental research (random sampling, control group vs. experimental group)May be used at any time in the research process for varying purposes Identify trends over time for particular populations Usually involves both qualitative and quantitative data (mixed-methods) data analysis 6

The Survey Instrument Is designed for particular purposes Is written in an understandable way (“standard language”); if in a foreign language, achieved by a professional translator or native speaker (not machine-translation, which is still pretty wretched) Uses close-ended questions appropriately with a full range of choices (no false limits)If scaled responses, proper scaling (like Likert-like scales and consistency of order; or forced-choice 4-point Likert scales with no fence-sitting neutrality); if scaled responses, proper consistency in terms of direction (highest-to-lowest for all questions; or lowest-to-highest for all questions) Uses open-ended questions appropriately, with sufficient direction and space for a full textual response Is accessible for all those with a range of special and other needs (transcriptions and timed text for videos, alt text for images, etc.) 7

The Survey Instrument (cont.) Aligns the questions with the appropriate data types [categorical, ordinal (rank order), numerical (discrete, continuous), text- / audio- based / video-based , and others )Includes informed consent at the beginning; enables opt-out at any time; no collection of excess information; no deception (unless approved by the Institutional Review Board / IRB)Is informed by the research literature (explored to “saturation”) Is strategically sequenced Avoids forcing responses because of the participant opt-out issue per IRB guidelines (debatable) 8

The Survey Instrument (cont.) Avoids any biasing design or leading language Is pilot-tested with both experts and with people who are similar to respondents, with changes made to ensure language clarity; comprehensiveness of the survey; clear transitions; accessibility; and corrections of all known errors (and continuing testing until no other errors are found) Is tested for reliability (that it is dependable and consistent) Is tested for validity (that it measures what it purports to measure) Aligns with domain’s professional research standards and expectations May be versioned for different groups, or may be branched for certain groups Must maintain comparability if studied for trend data for longitudinal research 9

Survey Reliability Achieving the same results every time the instrument is used, such as through test-retest reliability with the same person or group (over time) Reliability across different instruments or “equivalence reliability” Internal consistency of measure [Cronbach’s alpha α / coefficient alpha; the complementarity of questions in relation to each other in measuring one dimension or a single construct (unidimensional); inter-correlations among the test items; not robust under conditions of missing data; variables and the degree to which they measure the same thing in an inter-item correlation way as expressed in a matrix and comparisons done by removing variables to see what changes occur in the measuring of the construct; a latent construct may affect the alpha; α < 1 ] 10

Survey Validity Accurate measurement of what it was designed to measure Different types of validity: Predictive validity Concurrent validity (against an accepted measure)Content validity (reasonable sample of related information and proper terms for what the survey wants to sample) (Fink, 2013, p. 67) Construct validity (using the instrument on respondents who’ve been established by experts to rate a particular way on a particular scale on a particular construct to see if the target survey comes up with the same results) 11

The Credibility of Survey Findings “Reliability” and “validity” are developed to support the “measurement validity” and ultimately the credibility of survey findings Also need to control for “error,” which comes from many sources: Representative sampling, eligibility criteria of those taking the survey, low response rates, attrition of participants (particularly in longitudinal research) Researcher effects: cognitive biases, incentives, weaknesses Research designInstrument design Administration F ollow-on survey without sufficient passage of time (and the effect of the first survey’s results on the latter)Exclusion / inclusion of outlier data pointsInsufficient analysis and refinement 12

The Time Factor Cross-sectional or slice-in-time surveying Multiple-sequential surveying Longitudinal (or periodic over-time surveying) 13

Sampling of Survey Respondents Random (and sufficient) sampling the “gold standard” for generalizing to a population Stratified random sampling to select members of particular groups as respondents Simple random cluster sampling (convenience sampling, assumption of pre-defined clusters in the population) Convenience sampling (like snowball sampling); non-random; gold standard as “representative” sampling for qualitative research Systematic sampling (like every 5 th person…, may have hidden if unintentional biases, with a common example as A-Z sampling but with fewer individuals with names in the W-Z range) 14

Sampling of Survey Respondents (cont.) Open-call sampling with an online survey Bias in terms of those who self-select or opt-in, have techno access and techno savvy Potential difficulty in verifying identity P otential broader geographic reach Case control: case group (“extant” condition) and control group (absence of “extant” condition) for comparison and contrast and potential generalizing 15

Online Surveys Researcher needs to know and deploy the technology well Must protect the data well to meet all legal guidelines (going with a trusted survey company) Must protect participant privacy and confidentiality Must de-identify data / anonymize before data ingestion into an analysis tool (or dataset sharing through repositories or “reproducible research” articles) Must offer opt-out function at any time (for IRB standards) Must anticipate potential harm and mitigate Online survey m ust be fully comprehensible without survey taker intervention (designed to head off potential misinterpretation with additional opt-in data as needed) Data usually a mix of quantitative and qualitative dataMay be exported as .csv, .docx , .pdf, and other file types May be partially exported in pre-made tables, charts, and graphs 16

Data Forms . xlsx data tables (for quant data) .csv text files, .doc and .docx text files Some pre-extracted bar charts from the online survey systems 17

Interviews 18

Basic Purposes of Interviews Targeted elicitation of information Achieving specific insights Applied for various practical purposes: awareness, research, decision-making, vetting, historical understandings, and others 19

Types of Interviews Structured, semi-structured, or unstructured (pre-written and non-changing…to play-it-by-ear) Formal or informal Individual or group On-the-record, off-the-record (information usable but not publicly quotable or attributable back to the source) May be conducted in the field; may be in a more controlled setting; may be conducted online 20

Data Forms Videos, audios, and derived transcripts I nterviewee-created materialsResearcher notes, memos, and other recorded materials …and others… 21

Focus Groups 22

Basic Purposes of Focus Groups The elicitation of information from a homogeneous group (homogeneous based on a relevant dimension) targeted because of their access to particular types of information in a session or multiple sessions conducted by a non-directive interviewer (who uses sequenced questions, activities, and nuanced elicitation to acquire responses beyond intellectualized ones—such as emotions and unconscious behaviors) Used for community outreach, brand testing, product testing, problem-solving 23

Seating the Focus Group Usually 8 – 20 people, with preference towards fewer for manageability and coherence Homogeneous group based on a particular dimension (such as access to experiences and / or categorical biographical features); convenience sampling; quasi-experimental Focus on comfort level of members in speaking and sharing May divide focus groups into different sexes, for full encouragement of information sharing Seat individuals who will likely not see each other again Avoid power differentials in the group (because of reactivity to power) If cross-cultural context, many efforts for cultural sensitivity with information from informants and native speakers May have to “over-sample” for wider geographic reach (Krueger & Casey, 2013, p. 27) 24

Complex Researcher Role Researcher: “moderator , listener, observer, analyst” (Krueger & Casey, 2013, p. 7)Encourages all to speak early on, so some do not lapse into silence for the session (Krueger & Casey, 2013, p. 39) Does not share his or her opinion or any biasing expressions or body language Must be able to handle high affect or emotions (if the research calls for stressors) Must handle dominant personalities Must encourage variant opinions Must be comfortable with serendipity and interaction effects among the participants Should “reality check” understandings with participants during the session (Krueger & Casey, 2013, p. 189) 25

Information Elicitation Methods Must design test questions and elicitations (like prompts or activities) that result in usable data Must generally include a catch-all question at the end to make sure nothing was missed Must design a “questioning route” or sequence that makes sense to participants Should ask questions answerable by the particular group Should run positive questions first and negative ones later Should run simpler questions first and more difficult ones later Must design activities that get at more elusive information (such as emotions or unconscious ideas) Should pilot-test draft questions and dry-run processes (and pacing) with individuals most similar to those who would likely take part in the actual focus groups 26

Data Collection Methods Face-to-Face (F2F) Video-recording (may be unobtrusive and through one-way mirrors), audio recording, human coders, participant writingMay be captured by computers if used in the session Researcher observations (and journaling or memo-ing) Online Synchronous and live (facilitated); asynchronous (may be facilitated or non-facilitated) Web session recording Transcription (full or abridged) 27

Data Issues “Verbal exchange coding” may be one approach to coding the data (Saldaña, 2013, pp. 136 – 141) Researchers cannot assert beyond where the information will go. They…cannot generalize from the findings. cannot stereotype participants to “stand in” for others of the same racial background, ethnic background, demographic group, social class, etc. should describe the research in sufficient detail so that it is theoretically and practically replicable. should avoid any potential mis-use of the results or findings.should use the research to benefit participants. 28

Some Limits Difficulty in finding individuals who fit a particular requirement (characteristic or experience) Quasi-experimental study does not result in generalizable results Challenges in setting up incentive structures to encourage participation Need to protect participant privacy Must run focus groups as many times as needed to get to the desired data (to the point of “saturation,” p. 21) Costs of hosting focus groups If working with community partners, need to ensure that they also follow IRB guidelines and other policies and laws at K-State 29

Data Forms Transcripts Videos Audio filesNotes Participant-created artifactsParticipant performance 30

The Data and Nvivo 10 31

Parametric Data Assumption of a Gaussian “bell curve” as the underlying data structure Need for statistical significance Ability to reject the null hypothesis Univariate data: descriptive statistical measures (of a population based on findings from a random or stratified random sample), such as arithmetic mean (central tendency), median, standard deviation (statistical dispersion), variance, min-max (range), confidence interval (for single or repeated sampling), and others Inferential or inductive statistics 32

Parametric Data (cont.) Bivariate data: If conducting a linear regression to compare an independent variable (IV) and a potential association with a dependent variable (IV), you’ll need paired data with the IV on the x-axis and the DV on the y-axis, the paired data plotted, and a line drawn based on the least-squares method…and a study of the “scatter” of the plotted data, whether a line exists, whether a line is linear or curvilinear, whether the line slope is positive or negative, and so on Pearson product-moment correlation (Pearson’s r): analysis of associations between two variables (based on SDs from the expected mean); co-variance of two variables divided by the product of their standard deviations; measures linear dependence between two variables; “the standardizing of covariance between -1 and 1” 33

Parametric Data (cont.) Multivariate data: If conducting a factor analysis (from survey data), minimize factors with a principal components analysis (a form of data reduction) and to identify potential multi- collinearity ANOVAs (analyses of variance) MANOVAs (multivariate analysis of variance), and others 34

Non-parametric Data Assumption of discrete data range and expected average as the underlying data structure Ability to assert an observable effect beyond chance Chi-square test (table) and the calculation of the expected values if there is no effect (or the likelihood of the categorical data spread based on chance alone, or the probability based on just If sample size too small for a t-test, the Mann-Whitney U test ( Wilcoxian rank sum ), with data compared against what one would find based on alternative and competing hypotheses 35

Non-parametric Data (cont.) Spearman’s rho (rank order correlation) for categorical data that is set up on an ordinal (rank) scale (two sets of non-parametric ranks) Pearson’s r “bootstrap” for qualitative data: “Non-parametric statistics.” 36

Content Analysis (for Textual Data) Extant themes* Use of language (“in vivo” coding by using the language of the respondents for codes) Direction and strength of sentiment; “emotion coding”; attitudes and beliefs Causal attribution analysis* Comparisons across categories of cases / individuals (for theoretical generalizability)Quotable insights (to use in the “reporting out” phase)* * Common “generic” elements that are extracted 37

Content Analysis (for Textual Data) (cont.) Semantic analysis: strict inclusion; spatial; cause-effect; rationale; location for action; function; means-end; sequence; attribution ( Spradley, 1979, as cited in Miles, Huberman, & Saldaña, 2014, p. 179) Display-based analyses: matrix analyses, network analyses 38

First and Second Cycle Coding First Cycle (Initial) (aka the coding nodes) DescriptiveIn vivo codingProcess coding Emotion codingValues coding Evaluation coding Dramaturgical coding Holistic coding Second Cycle (Follow-on) (aka “Pattern Codes,” the organization of the coded nodes) Categories or themes Causes / explanations Relationships among people Theoretical constructs (Miles, Huberman , & Saldaña , 2014, p. 87) 39

First and Second Cycle Coding (cont.) First Cycle (Initial) Provisional coding Hypothesis coding Protocol coding Causation coding Attribute coding Magnitude coding Subcoding Simultaneous coding (Miles, Huberman, & Saldaña, 2014, pp. 74 – 81) Second Cycle (Follow-on) 40

Reporting Out: Proper Data Representation Correct data representations (text, visualizations, videos, and interactive pieces) Clear labels on all visualizations Explanatory legends Lead-up and lead-away texts explaining the data visualizations (efforts to head off negative learning or misunderstandings from the data; controlling for false inferences) “Triangulation” of data from various sources Considering competing explanations for the observed data Proper qualifications of the data Higher standards for assertions of predictivity Acknowledged point-of-view (POV) of the researcher 41

“Surveys and Interviews in NVivo ” Demos 42

DEMO: Pre-Ingestion into NVivo Transcription of audio and video files Machine-based transcription with human review and double-check; manual coding with review and double-check Keeping original voice, verbiage, syntax, and grammatical mistakes (verbatim); acknowledging incoherence of the audio or video (full or abridged transcription) Multilingual challenges Annotation of imagery Cleaning up of memos, notes, and research journal entries Consistent file naming protocols across all files Within-file annotations and labeling 43

DEMO: Pre-Ingestion into NVivo (cont.) Data cleaning Aligning units of measure, units of time, and any other relevant units Data disambiguation Content moving for different file types (. docx, .xl, .csv, and others) Deletion of unreconcilable information Data structuring / semi-structuring Early matrix or table or categorization work Early ideas for coding (based on theory) Data log Planning also for potential data ingestion into other data-analytic software tools and applications in addition to NVivo 44

DEMO: Pre-Ingestion into NVivo (cont.) De-identifying data and then ingesting the data into Nvivo (for participant privacy protections) Not before the identified data has been fully exploited (not “ lossiness” with usable data) Not in a re-identifiable (reverse-engineering from summary data) way Separate file for “encrypting” of coded identities (not in the . nvp file), so the researcher can re-identify the data if needed Erasure of metadata riding on digital files: videos, images, Word files, and others Data managementAccess only to those who should have access (“principle of least privilege”) Proper data storage for the length of promised time (three years?) Offline storage and encryption (very slow) if highly sensitive Proper handling if published as part of “reproducible research” or released / published datasets 45

DEMO: Ingestion Into Nvivo (cont.) In NVivoOne comprehensive project or many smaller (related) ones Coherent folder structures Setup for group coding (if applicable) Autocoding by text style (tagging) Creating question nodes for summary-by-questions and then manual copying of contents into the respective nodes Creating respondent nodes for matrix coding queriesManual coding Creating demographic groupings through Person Node Classifications (and pre-set and / or customized descriptive attributes) for cross-group comparisons 46

DEMO: Data Queries and Processing In Nvivo (cont.) Matrix codingBased on nodesBased on Person Node Classifications Data queries Text frequency counts and word clustering (and related data visualizations) Text searches and related word trees (and related data visualizations) … and others … 47

DEMO: Outputs from Nvivo (cont.) Data visualizations Word clouds, graphs, and cluster diagrams (2D, 3D) (.jpg, .jpeg; .bmp, .gif, and .pdf) Data tables (.xlsx) Data matrices (. xlsx)Formatted reports (. nvr as a transferable format between NVivo projects; .txt, .docx, .rtf, .pdf, .xlsx., .xls , and .htm / .html) …and easy transcoding from Excel workbook or worksheet to .csv, .xml, .txt, .pdf, and others; visual formats to . png , . tif , and others 48

References Fink, A. (2013). How to Conduct Surveys: A Step-by-Step Guide. Los Angeles: SAGE Publications. Krueger, R.A. & Casey, M.A. (2013). Focus Groups: A Practical Guide for Applied Research. (4th ed.) Los Angeles: SAGE Publications. Miles, M.B., Huberman, A.M., & Saldaña , J. (2014). Qualitative Data Analysis: A Methods Sourcebook. (3rd Ed.) Los Angeles: SAGE Publications. Saldaña, J. (2013). The Coding Manual for Qualitative Researchers. (2nd Ed.) London: SAGE Publications. 49 Note : These texts will be available for perusal after the presentation.

Conclusion and Contact Info Dr. Shalin Hai-Jew Instructional Designer, iTAC K-State 212 Hale Library785-532-5262 shalin@k-state.edu (Note : Thanks to Alice Anderson for patiently critiquing an early version of this. A few changes were made, and I decided to make this a post-presentation handout instead of a pre-event refresher.) Resource Using NVivo: An Unofficial and Unauthorized Primer 50