Designing & Implementing Effective Surveys Emily Daly & Joyce Chapman - PowerPoint Presentation

Designing & Implementing Effective Surveys Emily Daly & Joyce Chapman Duke University Libraries State Library of North Carolina September 2017

IntroductionsEmily Daly, Head of Assessment & User Experience Joyce Chapman, Assessment Analyst & Consultant

AgendaQuestions before we get started?Basic survey structure Writing actionable survey questions Survey tools (briefly!) Acting on survey dataKey takeaways

Time to hear from you!Which methods do you use to distribute surveys in your library?

PollWhich methods do you use to distribute surveys in your library? ( enter your answers in the chat box – e.g., a, c, d) Paper surveys, distributed by mail Paper surveys, distributed in your building or by hand Electronic (web-based) surveys Telephone surveys In-person surveys Other (please describe in chat)

STRUCTURE OF THE survey

Form follows functionWhat are you most interested in learning?Who will analyze the data? How? Who will see the results? Who is your target response group? How much time do they have? How invested are your respondents?

The introductory statement

Some examples“The Duplin County Public Library is interested in learning more about what community members think of services provided by the library .”

Some examples“The Duplin County Public Library is interested in learning more about what community members think of services provided by the library .” “The purpose of this study is to identify areas undergraduate students feel should be addressed in order to maintain an effective academic library.”

More examples“The library is particularly interested in the opinions of patrons who live in the western part of the county. You have been selected at random from community residents in western Greene County .”

More examples“The library is particularly interested in the opinions of patrons who live in the western part of the county. You have been selected at random from community residents in western Greene County .” “ Your opinions are very important to us. There are no right or wrong answers. Your responses will be treated confidentially. Survey results will in no way be traceable to individual respondents.”

Question sequence

Strategies for good sequenceStart with well formatted, engaging questionsDon’t lead with questions on sensitive topics Place questions you care about most in the first half Avoid repetitive, consecutive questions that lead to reflexive responses Consider ending with demographics questions

Status updates

Image: www.clustershot.com/garysimmons/photo649731

Image: https://flic.kr/p/coyARS

Closed responsesAdvantagesUniform response set facilitates comparison, choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed

Closed responsesAdvantagesUniform response set facilitates comparison, choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed Disadvantages Random selection, “closest representation” issue, loss of distinction

Open responsesAdvantagesAllow for deep explanations, only way to get responses you wouldn’t otherwise know

Open responsesAdvantagesAllow for deep explanations, only way to get responses you wouldn’t otherwise know Disadvantages Requires communication skills from respondents, more time consuming for respondent and analysis

Compromise Would you recommend this search results screen to a friend or colleague? Yes Maybe No

Compromise Would you recommend this search results screen to a friend or colleague? Yes Maybe No Why, or why not?

Last call for feedback!

Last call for feedback! Any additional comments about the Natrona County Public Library?

Last call for feedback! Any additional comments about the Natrona County Public Library? What else would you like to tell us about your experience using this page?

Last call for feedback! Any additional comments about the Natrona County Public Library? What else would you like to tell us about your experience using this page? Is there anything else you’d like to share?

Closing your surveyDemographics questions for data you can’t get elsewhere Call for volunteersClear SUBMIT button Thank respondents for participating! Web-based: Redirect to a related page or the library homepage

Invitations to participate“Would you be willing to participate in future discussions or focus groups about the library? If so, please provide your contact information below .” “Would you be interested in joining a community of makers to present their work in the Maker Space?” Yes, I'm interested in being contacted about this opportunity No, I'm not interested in being contacted about this opportunity

Questions or comments?

Writing unbiased, actionable, effective questions

Avoid jargon and colloquialismsThink of your audience. Wording should be simple. Avoid technical words. X "How many times last week did you use the library's Internet-enabled public access computers ?"

Avoid jargon and colloquialismsThink of your audience. Wording should be simple. Avoid technical words. X "How many times last week did you use the library's Internet-enabled public access computers ?" O "How many times last week did you use the library's computers [to access the Internet] ?"

Abbreviations and acronymsAssume your audience does not know any of these. X “CCPL is interested in replacing its OPAC." X “How frequently do you use our ILL services?”

Abbreviations and acronymsAssume your audience does not know any of these. X “CCPL is interested in replacing its OPAC." O “ Cleveland County Public Library is interested in replacing its online catalog .” X “How frequently do you use our ILL services?” O “How frequently do you use our Interlibrary Loan services ? [This service allows us to request material from another library for you if we do not have it here].”

Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings? X "What is your income?" X "How many people are there in your household ?"

Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings? X "What is your income?" O "What is your income before taxes? Include salary as well as other sources of income. " X "How many people are there in your household?" O " Including yourself, how many people are there in your household ?"

Confusing phrasingThe respondent should not have to spend time re-reading/interpreting the question. X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?" It seems likely __ It seems unlikely __ I’m not sure __

Confusing phrasingThe respondent should not have to spend time re-reading/interpreting the question. X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?" It seems likely __ It seems unlikely __ I’m not sure __ O "If the library had a Maker Space, would you use it?" Yes__ No__ Unsure__

Avoid double barreled questionsA question that introduces two or more issues with the expectation of a single response X "Is our staff friendly and professional?"

Avoid double barreled questionsA question that introduces two or more issues with the expectation of a single response X "Is our staff friendly and professional?" O Question1: "Is our staff friendly?" O Question 2: "Is our staff professional ?"

Avoid non-specific questionsDo not leave questions open to a wide range of interpretations. X “How do you feel about public transportation?”

Avoid non-specific questionsDo not leave questions open to a wide range of interpretations. X “How do you feel about public transportation?” O “How do you feel about the DATA bus system in Durham County, North Carolina?”

Manipulative informationCertain questions require some background . Be careful that explanatory statements do not unduly influence responses. X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15 per resident? "

Manipulative informationCertain questions require some background . Be careful that explanatory statements do not unduly influence responses. X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15 per resident? " O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per resident?"

Manipulative informationWe often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info. O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per resident?" O "If you were to learn that the county government spends approximately $10 per resident on landscaping public areas, would that change your opinion about the adequacy of allocating $1.15 per resident to the library?"

Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don 't assume answers at the top are more important to the interviewer, or have software randomize them. X Group study rooms Digital media lab Laptop lending E-books Printing/copying

Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don 't assume answers at the top are more important to the interviewer, or have software randomize them. X Group study rooms O Digital media lab Digital media lab E-books Laptop lending Group study rooms E-books Laptop lending Printing/copying Printing/copying

Interval categoriesDo not allow to overlap. Provide an unbounded final category if appropriate. X Age 0-10 Age 10-15 Age 15-20 Age 20-50 Age 50-75

Interval categoriesDo not allow to overlap. Provide an unbounded final category if appropriate. X Age 0-10 O Age 0-9 Age 10-15 Age 10-19 Age 15-20 Age 20-29 Age 20-50 Age 30-39 Age 50-75 Age 40-49 Age 50+

Multiple response clarificationSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear. X For which of the following reasons do you use the library? X__ Y__ Z__

Multiple response clarificationSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear. X For which of the following reasons do you use the library? X__ Y__ Z__ O For which of the following reasons do you use the library? Choose all that apply. X__ Y__ Z__

Appropriate response choicesSurveys can be frustrating when the questions are fixed response without appropriate answer choices. P rovide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate. X Does the laptop lending program meet your needs? Yes__ No__

Appropriate response choicesSurveys can be frustrating when the questions are fixed response without appropriate answer choices. P rovide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate. X Does the laptop lending program meet your needs? Yes__ No__ O Does the laptop lending program meet your needs? Yes__ No__ N/A__ [or “I’ve never used this program__”]

Questions or comments?

Survey tools

Time to hear from you!What survey tools have you used?

PollWhat survey tools have you used? (enter your answers in the chat box – e.g., a, c, d) Google Forms Paper-based Poll Everywhere Qualtrics Survey Monkey Other (let us know in the chat box!)

Survey toolsThere are many tools! I’ll mention a few, feel free to share your experiences in the chat Google Forms Paper! Qualtrics Survey Monkey

Google forms

Google forms

Google forms

Qualtrics

Qualtrics

Survey Monkey

Survey Monkey

Paper surveys Image: http:// www.educationnews.org/wp-content/uploads/2015/08/paper.jpg

Questions or comments?

Acting on survey data

Involve your colleagues Image: http://www.flickr.com/photos/dukeunivlibraries/

Share your resultsShort email with key points about the resultsPresentation for key staff or departments or even the entire libraryBrief report, sent only to the staff most interested in the results Links to the data for staff to explore on their own

Code free text responses

Crowdsource the work!

Outreach opportunities“Funding for students to purchase articles that Duke Libraries doesn't have access to would be nice. Often I find current articles that I can use for research but they are over $60.” “It would be nice to be able to return Lilly Library DVDs to Perkins & Bostock .”

Triangulating dataMethods to consider when triangulating data:thoughtfully planned focus groupssemi-structured interviews observational studies targeted, more focused surveys u sage statistics or other numerical data (e.g., gate counts, circulation stats, web metrics)

Follow-up focus groups Image: http://www.flickr.com/photos/dukeunivlibraries/

Reports, presentations, dashboards Tableau dashboards: bit.ly/SurveyDashboards-2016

Staff workshop Chart : emba.mit.edu/images/uploads/Impact_Matrix_MIT_EMBA.jpg Image: http://www.flickr.com/photos/dukeunivlibraries/

Staff workshop Chart : emba.mit.edu/images/uploads/Impact_Matrix_MIT_EMBA.jpg Image: http://www.flickr.com/photos/dukeunivlibraries/

Organizing all of the great ideasPotential expendituresService improvementsMarketing opportunities Assessment opportunities

…and following through

Sample projectsLibrary e-newsletter Dry erase marker kits Low-cost earbuds Posting hours to the coffee shop Improved signage FitDesks

Process from start to finish Conduct survey Analyze data Share findings Develop recommendations Prioritize according to ease of implementation and potential impact Assign ownership Follow through

Questions or comments?

Key TakeAways

Involve your colleagues… Image: www.flickr.com/photos/dukeunivlibraries/

and users… Image: www.flickr.com/photos/dukeunivlibraries/

Form follows function

Plan to analyze

Test early and often

Triangulate methods Image: https://commons.wikimedia.org/wiki/File:Japanese_Map_symbol_(Triangulation_point).svg

Act on what you learn Image: http://www.flickr.com/photos/dukeunivlibraries/

Thank you!The State Library will send participants a link to the recorded workshop and slide deck. With questions about our content, contact Emily Daly: emily.daly@duke.edu Joyce Chapman: joyce.chapman@duke.edu With questions for the State Library, contact Amanda Johnson: amanda.johnson@ncdcr.gov Lauren Clossey: lauren.clossey@ncdcr.gov