/
Introduction to  Survey Research Introduction to  Survey Research

Introduction to Survey Research - PowerPoint Presentation

mediumgeneral
mediumgeneral . @mediumgeneral
Follow
346 views
Uploaded On 2020-06-17

Introduction to Survey Research - PPT Presentation

Sampling Questions Methods and Dissemination of Findings Agenda 1 Introduction 2 Introduction to Survey Sampling 3 Questionnaire Design 4 Methods Overview 5 Best Practices for Public Dissemination of Findings ID: 779679

survey sample design sampling sample survey sampling design population questions question rules thumb cont

Share:

Link:

Embed:

Download Presentation from below link

Download The PPT/PDF document "Introduction to Survey Research" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Introduction to

Survey ResearchSampling, Questions, Methods, and Dissemination of Findings

Slide2

Agenda

1. Introduction

2. Introduction to Survey Sampling

3. Questionnaire Design

4. Methods Overview

5. Best Practices for Public Dissemination of Findings

Slide3

Introduction

Slide4

Why do we conduct surveys?

To collect data that allow us to understand how and why specific

populations

behave and think about topics of interest.

Surveys can complement findings from qualitative studies by allowing us to size populations by attributes, opinions, behaviors – and track them over time.

Slide5

Slide6

Slide7

Slide8

Introduction to Survey Sampling

Slide9

What makes a successful survey?

Many things: appropriate, clear, and non-biasing questions; well-trained interviewers; appropriate data collection method; etc. But first, we must be confident we are surveying a sample of people representative of the population of interest.

Slide10

What is sampling, and why does it matter?

Much of what survey researchers do is an art: writing questions, interacting with people during interviews, making sense of the data after the interviews are completed.But sampling is what makes survey research a science.

Good sampling allows us to measure attributes of the target population within known statistical parameters using probability theory. If the study does not use a probability sample, it is not a survey.

Slide11

Two ways we can make sure we are talking to a representative sample

1. We can talk to everyone in the target population. This is called a census.

2. We can talk to a representative

subset of the population. This is called a probability sample.

Slide12

Pros and cons of conducting a census

Pros: Do not have to worry about talking to an unrepresentative sample; can measure very small differences between subgroups (no sampling error).

Cons:

Nearly impossible to conduct; cost is prohibitive; takes ages to do.

Slide13

Pros and cons of conducting a survey

Pros: Possible to do (though increasingly difficult), costs less than a census; takes less time than a census.Cons: Difficult to achieve a representative sample, must be done correctly; more difficult to detect subtle differences between subgroups (due to sampling error).

Slide14

“How can you tell what a whole country is thinking by talking to only a thousand of its citizens?”

We don’t need to eat an entire pot of soup to know what it tastes like. We just need to taste a spoonful. But we have to make sure we get a representative spoonful.

Slide15

Probability samples

A probability sample is a sample where every person in the target population has a known, non-zero probability of being selected. That is, it is possible for every person in the target population to be in the sample. Two important features: 1. Guards against selection bias on all variables – sex, age, region, political views, eye color, everything.

2. The basis for scientific inference about characteristics of the entire target population.

Slide16

What does that mean – the basis for scientific inference about characteristics of the entire target population?

It means that if we use probabilistic sampling methods, we can be confident our estimates describe the true target population values within a margin of error because we can quantify, based on probability theory, the uncertainty resulting from sampling error.

In other words, probability sampling allows us to make statements about the target population with a measurable degree of confidence.

Slide17

Sampling error

Random error occurs when a particular sample is not representative of the target population due to random variation. It can be expressed as the difference between the sample results and the true results.

Even if all aspects of the sample are executed properly, the results are still subject to a certain amount of error due to random, chance variation.

Slide18

Margin of error

A margin of error quantifies the uncertainty resulting from sampling error in our survey results. Margins of error allow us to estimate from a survey statistic the true population value within a value range specified by a confidence level (usually 95%). For example:

70% believe global warming is happening, +/- 3 percentage points at a 95% confidence level.”

This means we are 95% confident that the true population value is between 73% and 67%. Specifying a margin of error is appropriate only with

probability samples.

Slide19

Good common sampling methods

Random digit dialingRandom digit dial (RDD) and in-person area probability samples are best – and the most expensive. Cell phone numbers should comprise 40% of the sample, and this number will grow.Online panels maintained by Knowledge Networks, NORC, RANDProbabilistically drawn panels are good as well, but recognize that panel members represent only a very small proportion of those initially recruited to join the panel (2-5%) and therefore have unknown biases.

Slide20

Non-probability sampling procedures

Quota sampling: quotas usually not filled on a random basis; often relies on judgment of interviewer/sampling statistician.

Convenience sampling:

“typical” cities; intercept interviews; panels; lists.

Judgment sampling:

key informants, customers; early adopters; opinion leaders.

Snowball/network sampling:

rare populations.

Slide21

Caution: A sample is only as good as the sampling frame

The sample frame is the list, real or theoretical, from which we select units for our sample.

It is desirable for the sampling frame to match the target population and avoid frame errors associated with omitting target population units or including units outside the target population.

Slide22

Caution: Drawing a perfect sample doesn’t mean you always survey a perfect sample

For example, one nagging and growing problem is declining response rates. If we can

t reach many of the people we sample, and then only a few of those we do reach agree to take the survey, are we ending up with a probability sample? No. Our sample would comprise only respondents we could reach and who agreed to take our survey, hardly a representative sample, hardly a well-defined sample frame.

Slide23

Question

What is the value of conducting a survey to learn something about a population that does not use a probability sample – whether in academia, business, or government?

Slide24

Questionnaire Design

Slide25

Questions should be designed to produce

“need to know” information that advances your specific goals. For example:The Yale Program on Climate Change Communication…

* Measures current key beliefs and tracks them over time: Whether or not GW is happening, human-caused, has serious consequences, solvable, scientific consensus, etc.

* How do different “segments”

of the population think about climate change? How big are the receptive (and skeptical) segments? Are they growing or shrinking? Why?

* How can communicators help move people up the ladder of engagement or from one segment to another?

Determine Which Questions Need to Be Answered

Slide26

* What messages work best with different segments? Which ones work across segments? Which ones alienate certain segments?

* Who are the most effective messengers? In which situations?* What are the best ways to reach receptive audiences (e.g., via media, clergy, word of mouth, etc.).* What is the most effective creative execution of the messaging?* What are the common and unique challenges in other countries and cultures?

Determine Which Questions Need to be Answered (Cont

d.)

Slide27

Have these questions been asked before?

By whom and in what context? What was the result? Does prior research answer our question(s) or require more insight? Are there other surveys we can mine?* Check the academic literature (Nexus/Lexis, POQ, appropriate academic journals)* iPoll (the Roper Center – largest repository of public opinion research in the world)* Web search

* Talk to other organizations – redundancy should be avoided

Background Research

Slide28

The Questionnaire

* Questions should be asked to test specific hypotheses, not to go on a “fishing expedition” (thereby increasing the chance of Type I errors). * Analytic plan should be determined prior

to questionnaire design.

* Ask only those questions you need to ask so as not to burden the respondent. Still, do your best to avoid “post- survey regret” – wishing you had asked a question you hadn

t.

* Ask demographic questions necessary to effectively weight the sample to approximate target population norms.

Construct Hypotheses and Design the Survey

Slide29

* Use best practices when designing questions (a BIG topic!). Still, recognize that despite the voluminous scientific literature on the

“right” and “wrong” way to ask questions, crafting good questions remains an art as much as a science.

* Replicate successful questions from other well-designed instruments (e.g., the Census, CPS) when appropriate.

* Always offer to give the survey in Spanish if a respondent prefers it.

29

Construct Hypotheses and Design the Survey (Cont

d.)

Slide30

Balance question choices:

In your opinion, is global warming happening or is global warming not happening?Not:

In your opinion, is global warming happening?

Question Design: A Few Rules of Thumb

Slide31

Rotate answer choices:

In your opinion, is global warming happening or is global warming not happening?Alternated with:

In your opinion, is global warming not happening or is global warming happening?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide32

Branching:

If the presidential election were held today, would you vote for Hillary Clinton, the Democratic candidate, or Jeb Bush, the Republican Candidate, or someone else?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide33

Branching (cont’d):

[If answered “Hillary Clinton,” ask/display:]Would you…Definitely vote for Hillary ClintonProbably vote for Hillary Clinton

[If answered “Jeb Bush,” ask/display:]

Would you…Definitely vote for Jeb Bush

Probably vote for Jeb Bush

33

Question Design: A Few Rules of Thumb (Cont’d.)

Slide34

Clear wording (awful):

Do you think it is possible or impossible that the Holocaust never happened?* It is possible the Holocaust never happened* It is impossible the Holocaust never happened

34

Question Design: A Few Rules of Thumb (Cont’d.)

Slide35

Clear wording (better):

Which statement comes closest to your point of view?The Holocaust definitely happenedThe Holocaust probably happenedThe Holocaust probably did not happen

The Holocaust definitely did not happen

(Not sure)

Question Design: A Few Rules of Thumb (Cont’d.)

Slide36

Use colloquial language (not):

Some people say that Congress should enact a “revenue neutral tax swap” that would reduce the annual taxes paid by all Americans, while increasing the amount they pay annually for energy (such as gasoline and electricity) by the same total amount. The purpose of this policy would be to reduce energy use. Would you support or oppose this policy?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide37

Use colloquial language (instead):

Some people say that Congress should change tax policy by reducing the taxes paid by Americans out of their paycheck, while increasing the amount they pay for energy (such as gasoline and electricity) so that, by the end of year, we have paid the same amount in taxes as we do now. The purpose of this policy would be to reduce energy use. Would you support or oppose this policy?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide38

Ask questions people can answer:

Which is the correct quote from James Joyce’s Ulysses?…I noticed him when I was tasting the butter so I took my time Bartell dArcy too that he

used to make fun of…

Or…I noticed him when I was tasting the butter so I took my time Bartell dArcy too that we

used to make fun of…

Question Design: A Few Rules of Thumb (Cont’d.)

Slide39

Don’t “test” people:

Based on anything you may have heard or on your own personal experience, what would you estimate is the average height of Americans?Not:What is the average height of Americans?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide40

Don’t ask agree/disagree questions:

Not:Do you agree or disagree with the following statement…President Obama is a good president.Instead:

In your opinion, is President Obama an excellent, very good, good, fair, or poor president?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide41

Anchor scales when possible:

Would you say your health is…Excellent, very good, good, fair, or poor[ROTATE WITH] Poor, fair, good, very good, or excellent?Not:On a scale from 1 to 5, where “1” means “poor” and “5” means excellent, what number would you rate your own physical health?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide42

Include a midpoint in scales:

ExcellentVery goodGood(Only) fairPoor

Question Design: A Few Rules of Thumb (Cont’d.)

Slide43

Open end questions – Useful when there is a very large number of possible answers:

What is the name of the store where you (or your household’s primary shopper) buys most of your food?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide44

Open end questions – Useful for qualitative findings or quotes:

In your own words, why do you consider yourself a Democrat (Republican)?

Question Design: A Few Rules of Thumb (Cont’d.)

Slide45

Dissemination of Findings

Slide46

Survey findings can be useful in many ways and can aid in policy, communications, and social-marketing campaigns.

Currently, our group’s four target audiences are:

The public, through clear messaging in the major media: Internet, radio, TV, print

Various NGO and for-profit friends who can use our information constructively (via our self-published reports, websites, etc.)

The White House, Congress, and local governments with short, smartly-constructed presentations and easily-distributed, nice- looking white papers

The academic community through peer-reviewed publications

46

Dissemination of Findings

Slide47

The American Association for Public Opinion Research (AAPOR) provides detailed guidelines for the public release of survey findings. The guidelines are to meant ensure survey findings are fairly presented (no

cherry picking

), sponsors are clearly identified, and sufficient methodological information is provided so other may replicate the study.

Full standards of disclosure may be found here:

http://

www.aapor.org

/Disclosure_Standards1.htm

47

Dissemination of Findings (Cont

d.)

Slide48

Introduction to

Survey ResearchSampling, Questions, Methods, and Dissemination of Findings