Overview
Method category: Evaluative market experiment
How to Use This in GLIDR
Close-Ended Surveys focus on quantifiable questions with a clear set of possible answers.
In GLIDR, you would run an Experiment focused on the ideas you want to validate with this survey and come up with a hypothesis of which option you expect to win. Then, you would get out of the building and conduct the survey with your users, and come back to enter the data as Evidence - Other. You can attach the raw data as an attachment and write up your takeaways, then in the Analyze phase of the Experiment, look back on the survey and decide if it validated or invalidated your ideas.
Learn more about each of those aspects of GLIDR:
---
Close-Ended Survey
Article excerpted from The Real Startup Book
In Brief
Closed-end surveys help converge on what's relevant in great detail, particularly to customers or prospects. They are designed to create structured quantitative data, which lends itself to statistical techniques. These surveys preselect a number of possible answers for each question. They also help explore categories of data, for example, to explore segmentation.
Helps Answer
What is the breakdown of client concerns/problems/preferences in terms of percent of all clients in a segment?
How X varies with Y, e.g., how many part-time students find advanced calculus challenging?
What patterns emerge over time? (If repeated over time)
Ranking questions like: What is the order of priority? Who/what is the best option?
What provides the most satisfaction?
Tags
Quantitative
Analytical
Convergent
Description
Closed-end surveys consist of closed-end questions only. These types of surveys are most useful for exploring "known unknowns." Typically, this means that:
You've tried exploring what you don't know.
You've already chosen direction, and
There are still holes in your knowledge.
For example, a startup founder who has achieved problem-solution fit and has performed some smoke testing around key "happy case" assumptions could use a closed-end survey to prioritize or discover other issues.
As the survey giver, you are interested in finding patterns in the answers. By focusing in depth of a particular group of people, you hope to uncover hidden patterns in the data you gather.
Here are a few examples of closed-end questions:
How do you feel on a scale of 1 to 10 (10 highest)?
Are you pregnant? yes/no
What's your blood type:ABABO
These surveys can be delivered offline (expert with clipboard), online (popup form), or as a hybrid (iPad at a conference).
Time Commitment
Varies significantly with the survey length and methodology chosen. It can take an hour to configure an exit survey on a website, or it can take weeks to perform a large-scale in-person survey and manually enter the data into a useful format.
How To
Be very clear what you want to learn up-front when designing the questionnaire. For example, a survey would have very different questions for each of these goals:Discovering user problemsImproving an existing product/UXKeep track of user satisfaction (as a proxy for referrals)Improve customer service
Usually this goal will feed from your current overarching goal in the business. Ideally write out the goal of the survey in one sentence.
Formulate questions you'd like respondents to answer. As a rule of thumb, only include questions that you think will result in specific actions. For example, how will you use the data you gather to inform your decisions when it comes to catering, lodging, transportation, registration, event activities/workshops, and speakers?
Types of questions include:
Dichotomous: Do you have private health insurance? yes/no
Likert-type scale: To what extent are you satisfied?100 percent75 percent50 percent25 percent0 percent
List of items: My favorite food group is:GrainsMeatFruit and VegetableMilk
Ordinal: Sort the following according to the order of importance to you:PriceSpeedCost
Matrix question: Please rate the following company divisions with respect to knowledge of MS PowerPoint on a scale of 1 to 5:MarketingProductOperationsSalesITFinanceCollect the answers. This can be done face-to-face, by phone, online, or via postal mail.Organize the answers into a useful format. Typically a spreadsheet is good enough to organize the data. Then it can be loaded into a statistical tool, like R, or a database for further inquiry.
Interpreting Results
Use simple statistical techniques like correlation, ANOVA, or regression for further insights. You can also hire a data scientist to help interpret your results.
Be sure that you have enough respondents to be able to use rigorous tools. Check your sample size as a whole, but also for each subgroup. If you don't have enough within a segment, you can't make a statistically informed statement about that group of people.
Based on your responses, check to see if any of the questions were too obviously worded or confusing.
Look at overall scores. Compare average answers to a benchmark or predetermined expectations.
Also look at the distributions of responses. Are they normally distributed? Skewed? A power law distribution (i.e. 80/20)?
Create a visual summary of the results.
Then take action!
Potential Biases
Avoid overly sensitive questions – you are unlikely to get a "true" response.
Avoid leading questions, which subtly prompt the respondent to answer in a particular way."Are you for or against an increase in tobacco tax rates?""Are you in favor of increasing tobacco tax rates in order to protect our children's health?" (Leading)
Take into account that your respondents will often not want to admit to unsavory or socially undesirable behavior or preferences, particularly if they don't feel safe or the results aren't confidential.
Using emotionally loaded content can predictably skew results to a "yes" or a "no" or cause the respondent to abandon the survey if they don't identify. For example, "Where do you enjoy drinking beer?" implies that the respondent enjoys drinking beer specifically and would yield unpredictable results at an AA meeting.
While conducting surveys, never ask people what they would like to pay for. Usually they lie or are simply unaware.
Field Tips
Keep questions simple on closed questionnaires. For example, avoid hypotheticals.
Ask (and learn) one thing at a time. You can check for correlation and causation later.
Allow the respondent the option of answering with "not appropriate"/"don’t know"/"have no strong feelings".
Got a tip? Add a tweetable quote by emailing us: realbook@kromatic.com
Case Studies
Got a case study? Add a link by emailing us: realbook@kromatic.com
References
Got a reference? Add a link by emailing us: realbook@kromatic.com