Sampling Questions Methods and Dissemination of Findings Agenda 1 Introduction 2 Introduction to Survey Sampling 3 Questionnaire Design 4 Methods Overview 5 Best Practices for Public Dissemination of Findings ID: 779679
Download The PPT/PDF document "Introduction to Survey Research" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Introduction to
Survey ResearchSampling, Questions, Methods, and Dissemination of Findings
Slide2Agenda
1. Introduction
2. Introduction to Survey Sampling
3. Questionnaire Design
4. Methods Overview
5. Best Practices for Public Dissemination of Findings
Slide3Introduction
Slide4Why do we conduct surveys?
To collect data that allow us to understand how and why specific
populations
behave and think about topics of interest.
Surveys can complement findings from qualitative studies by allowing us to size populations by attributes, opinions, behaviors – and track them over time.
Slide5Slide6Slide7Slide8Introduction to Survey Sampling
Slide9What makes a successful survey?
Many things: appropriate, clear, and non-biasing questions; well-trained interviewers; appropriate data collection method; etc. But first, we must be confident we are surveying a sample of people representative of the population of interest.
Slide10What is sampling, and why does it matter?
Much of what survey researchers do is an art: writing questions, interacting with people during interviews, making sense of the data after the interviews are completed.But sampling is what makes survey research a science.
Good sampling allows us to measure attributes of the target population within known statistical parameters using probability theory. If the study does not use a probability sample, it is not a survey.
Slide11Two ways we can make sure we are talking to a representative sample
1. We can talk to everyone in the target population. This is called a census.
2. We can talk to a representative
subset of the population. This is called a probability sample.
Slide12Pros and cons of conducting a census
Pros: Do not have to worry about talking to an unrepresentative sample; can measure very small differences between subgroups (no sampling error).
Cons:
Nearly impossible to conduct; cost is prohibitive; takes ages to do.
Slide13Pros and cons of conducting a survey
Pros: Possible to do (though increasingly difficult), costs less than a census; takes less time than a census.Cons: Difficult to achieve a representative sample, must be done correctly; more difficult to detect subtle differences between subgroups (due to sampling error).
Slide14“How can you tell what a whole country is thinking by talking to only a thousand of its citizens?”
We don’t need to eat an entire pot of soup to know what it tastes like. We just need to taste a spoonful. But we have to make sure we get a representative spoonful.
Slide15Probability samples
A probability sample is a sample where every person in the target population has a known, non-zero probability of being selected. That is, it is possible for every person in the target population to be in the sample. Two important features: 1. Guards against selection bias on all variables – sex, age, region, political views, eye color, everything.
2. The basis for scientific inference about characteristics of the entire target population.
Slide16What does that mean – the basis for scientific inference about characteristics of the entire target population?
It means that if we use probabilistic sampling methods, we can be confident our estimates describe the true target population values within a margin of error because we can quantify, based on probability theory, the uncertainty resulting from sampling error.
In other words, probability sampling allows us to make statements about the target population with a measurable degree of confidence.
Slide17Sampling error
Random error occurs when a particular sample is not representative of the target population due to random variation. It can be expressed as the difference between the sample results and the true results.
Even if all aspects of the sample are executed properly, the results are still subject to a certain amount of error due to random, chance variation.
Slide18Margin of error
A margin of error quantifies the uncertainty resulting from sampling error in our survey results. Margins of error allow us to estimate from a survey statistic the true population value within a value range specified by a confidence level (usually 95%). For example:
“
70% believe global warming is happening, +/- 3 percentage points at a 95% confidence level.”
This means we are 95% confident that the true population value is between 73% and 67%. Specifying a margin of error is appropriate only with
probability samples.
Slide19Good common sampling methods
Random digit dialingRandom digit dial (RDD) and in-person area probability samples are best – and the most expensive. Cell phone numbers should comprise 40% of the sample, and this number will grow.Online panels maintained by Knowledge Networks, NORC, RANDProbabilistically drawn panels are good as well, but recognize that panel members represent only a very small proportion of those initially recruited to join the panel (2-5%) and therefore have unknown biases.
Slide20Non-probability sampling procedures
Quota sampling: quotas usually not filled on a random basis; often relies on judgment of interviewer/sampling statistician.
Convenience sampling:
“typical” cities; intercept interviews; panels; lists.
Judgment sampling:
key informants, customers; early adopters; opinion leaders.
Snowball/network sampling:
rare populations.
Slide21Caution: A sample is only as good as the sampling frame
The sample frame is the list, real or theoretical, from which we select units for our sample.
It is desirable for the sampling frame to match the target population and avoid frame errors associated with omitting target population units or including units outside the target population.
Slide22Caution: Drawing a perfect sample doesn’t mean you always survey a perfect sample
For example, one nagging and growing problem is declining response rates. If we can
’
t reach many of the people we sample, and then only a few of those we do reach agree to take the survey, are we ending up with a probability sample? No. Our sample would comprise only respondents we could reach and who agreed to take our survey, hardly a representative sample, hardly a well-defined sample frame.
Slide23Question
What is the value of conducting a survey to learn something about a population that does not use a probability sample – whether in academia, business, or government?
Slide24Questionnaire Design
Slide25Questions should be designed to produce
“need to know” information that advances your specific goals. For example:The Yale Program on Climate Change Communication…
* Measures current key beliefs and tracks them over time: Whether or not GW is happening, human-caused, has serious consequences, solvable, scientific consensus, etc.
* How do different “segments”
of the population think about climate change? How big are the receptive (and skeptical) segments? Are they growing or shrinking? Why?
* How can communicators help move people up the ladder of engagement or from one segment to another?
Determine Which Questions Need to Be Answered
Slide26* What messages work best with different segments? Which ones work across segments? Which ones alienate certain segments?
* Who are the most effective messengers? In which situations?* What are the best ways to reach receptive audiences (e.g., via media, clergy, word of mouth, etc.).* What is the most effective creative execution of the messaging?* What are the common and unique challenges in other countries and cultures?
Determine Which Questions Need to be Answered (Cont
’
d.)
Slide27Have these questions been asked before?
By whom and in what context? What was the result? Does prior research answer our question(s) or require more insight? Are there other surveys we can mine?* Check the academic literature (Nexus/Lexis, POQ, appropriate academic journals)* iPoll (the Roper Center – largest repository of public opinion research in the world)* Web search
* Talk to other organizations – redundancy should be avoided
Background Research
Slide28The Questionnaire
* Questions should be asked to test specific hypotheses, not to go on a “fishing expedition” (thereby increasing the chance of Type I errors). * Analytic plan should be determined prior
to questionnaire design.
* Ask only those questions you need to ask so as not to burden the respondent. Still, do your best to avoid “post- survey regret” – wishing you had asked a question you hadn
’
t.
* Ask demographic questions necessary to effectively weight the sample to approximate target population norms.
Construct Hypotheses and Design the Survey
Slide29* Use best practices when designing questions (a BIG topic!). Still, recognize that despite the voluminous scientific literature on the
“right” and “wrong” way to ask questions, crafting good questions remains an art as much as a science.
* Replicate successful questions from other well-designed instruments (e.g., the Census, CPS) when appropriate.
* Always offer to give the survey in Spanish if a respondent prefers it.
29
Construct Hypotheses and Design the Survey (Cont
’
d.)
Slide30Balance question choices:
In your opinion, is global warming happening or is global warming not happening?Not:
In your opinion, is global warming happening?
Question Design: A Few Rules of Thumb
Slide31Rotate answer choices:
In your opinion, is global warming happening or is global warming not happening?Alternated with:
In your opinion, is global warming not happening or is global warming happening?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide32Branching:
If the presidential election were held today, would you vote for Hillary Clinton, the Democratic candidate, or Jeb Bush, the Republican Candidate, or someone else?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide33Branching (cont’d):
[If answered “Hillary Clinton,” ask/display:]Would you…Definitely vote for Hillary ClintonProbably vote for Hillary Clinton
[If answered “Jeb Bush,” ask/display:]
Would you…Definitely vote for Jeb Bush
Probably vote for Jeb Bush
33
Question Design: A Few Rules of Thumb (Cont’d.)
Slide34Clear wording (awful):
Do you think it is possible or impossible that the Holocaust never happened?* It is possible the Holocaust never happened* It is impossible the Holocaust never happened
34
Question Design: A Few Rules of Thumb (Cont’d.)
Slide35Clear wording (better):
Which statement comes closest to your point of view?The Holocaust definitely happenedThe Holocaust probably happenedThe Holocaust probably did not happen
The Holocaust definitely did not happen
(Not sure)
Question Design: A Few Rules of Thumb (Cont’d.)
Slide36Use colloquial language (not):
Some people say that Congress should enact a “revenue neutral tax swap” that would reduce the annual taxes paid by all Americans, while increasing the amount they pay annually for energy (such as gasoline and electricity) by the same total amount. The purpose of this policy would be to reduce energy use. Would you support or oppose this policy?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide37Use colloquial language (instead):
Some people say that Congress should change tax policy by reducing the taxes paid by Americans out of their paycheck, while increasing the amount they pay for energy (such as gasoline and electricity) so that, by the end of year, we have paid the same amount in taxes as we do now. The purpose of this policy would be to reduce energy use. Would you support or oppose this policy?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide38Ask questions people can answer:
Which is the correct quote from James Joyce’s Ulysses?…I noticed him when I was tasting the butter so I took my time Bartell dArcy too that he
used to make fun of…
Or…I noticed him when I was tasting the butter so I took my time Bartell dArcy too that we
used to make fun of…
Question Design: A Few Rules of Thumb (Cont’d.)
Slide39Don’t “test” people:
Based on anything you may have heard or on your own personal experience, what would you estimate is the average height of Americans?Not:What is the average height of Americans?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide40Don’t ask agree/disagree questions:
Not:Do you agree or disagree with the following statement…President Obama is a good president.Instead:
In your opinion, is President Obama an excellent, very good, good, fair, or poor president?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide41Anchor scales when possible:
Would you say your health is…Excellent, very good, good, fair, or poor[ROTATE WITH] Poor, fair, good, very good, or excellent?Not:On a scale from 1 to 5, where “1” means “poor” and “5” means excellent, what number would you rate your own physical health?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide42Include a midpoint in scales:
ExcellentVery goodGood(Only) fairPoor
Question Design: A Few Rules of Thumb (Cont’d.)
Slide43Open end questions – Useful when there is a very large number of possible answers:
What is the name of the store where you (or your household’s primary shopper) buys most of your food?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide44Open end questions – Useful for qualitative findings or quotes:
In your own words, why do you consider yourself a Democrat (Republican)?
Question Design: A Few Rules of Thumb (Cont’d.)
Slide45Dissemination of Findings
Slide46Survey findings can be useful in many ways and can aid in policy, communications, and social-marketing campaigns.
Currently, our group’s four target audiences are:
The public, through clear messaging in the major media: Internet, radio, TV, print
Various NGO and for-profit friends who can use our information constructively (via our self-published reports, websites, etc.)
The White House, Congress, and local governments with short, smartly-constructed presentations and easily-distributed, nice- looking white papers
The academic community through peer-reviewed publications
46
Dissemination of Findings
Slide47The American Association for Public Opinion Research (AAPOR) provides detailed guidelines for the public release of survey findings. The guidelines are to meant ensure survey findings are fairly presented (no
“
cherry picking
”
), sponsors are clearly identified, and sufficient methodological information is provided so other may replicate the study.
Full standards of disclosure may be found here:
http://
www.aapor.org
/Disclosure_Standards1.htm
47
Dissemination of Findings (Cont
’
d.)
Slide48Introduction to
Survey ResearchSampling, Questions, Methods, and Dissemination of Findings