Overview
Method category: Evaluative market experiment
How to Use This in GLIDR
The Fake Door Smoke Test involves building a fake ad or button to gauge interest for a new feature or idea with users. Be sure to conduct the test ethically to maintain trust, and try to vary the way you present the button or ad to minimize the distorting impact of the copy and appearance.
In GLIDR, you should create an idea that there's demand for a particular feature, then start an Experiment to validate that demand. Once you've set up your plan, move the Experiment to Run and do the test through the channel of your choosing. Once it's complete, add the results of this test as Evidence. Finally, in the Analyze phase, reflect back on the results of the test to see if you hit your target metric for clicks to determine if there's sufficient demand for this feature.
Learn more about each of those aspects of GLIDR:
---
Fake Door Smoke Test
Article excerpted from The Real Startup Book
In Brief
People are bad at predicting what they want, but they are great at reacting to new offers. Building just the entry to an offer makes it possible to predict future reactions based on actual behavior. This test is also known as a 404 page test.
Helps Answer
Is the customer interested enough to click?
Which target audience is attracted?
Which words, icons, or images work well?
Tags
Testing
Quantitative validation
Positioning
Value proposition testing
Feature testing
Description
The fake door or 404 page test is a quantitative validation method that is useful in two circumstances:
To test if your idea sparks the interest of potential customers.
To test if an additional feature is a welcome addition to an existing product.
The idea of this test is to build a fake advertisement or button on a website to gauge the interest of potential users by looking at the clickthrough rate. The fake door could link to a “coming soon” page and prompt the user to add their email address to be notified when the product or feature is available.
Advantages
An idea can be quickly validated without spending a lot of time and money on development. Data is generated on users' actual behavior rather than asking them if they would be interested.
Ethics
Fake door testing can be thought of as tricking customers and decreasing the credibility of the product or business. When used carefully and in moderation, it can save time and money and lead to the development of a better product.
Clear communication that the feature or product is in development can prevent the user becoming upset that the product or feature isn’t available.
Internal and External Fake Doors
The internal fake door is great for testing if a potential new feature appeals to current users. Usually a link or button is created on a website for this new function.
One example would be a button on a local news website that says, “Switch to map view,” where the information is currently presented in a tile view. If users are satisfied with how the information is presented, it is likely they will not click this button. If users assume a map view would better suit their needs, it is likely they will click. The clickthrough rate (the amount of clicks divided by the number of views) will help to gauge user interest.
Be careful when interpreting results. For example, the location of a button can have more influence than the button itself. Always experiment with placing buttons in different locations.
Using an external fake door is a great test to see if your concept is interesting for a certain audience. A common way to test this is using a "fake ad." For example, say you have the idea to start an online community for Kingfisher enthusiasts. Before building a complete website, you set up a test and check if there are enough Kingfisher enthusiasts looking for an online community. You determine beforehand that if more than 5 percent of all views convert to a click, enough users are interested. Doing so, you can simply test if your idea suits the need.
Be careful. The text in the header and body of your ad have a great impact on how people react to the advertisement. Always try a variety of sentences in your headers and body of the ad to overcome this bias.
Fake doors are not suitable for testing your minimum viable product and essential functionalities, such as your registration process.
How To
Write down the hypothesis. Each test starts with writing down your key ideas in the form of a hypothesis.
Build the fake door(s). Make sure you would represent the feature in the same way as it would be built. If you exaggerate the button or particularly highlight the link, your results will be biased and therefore useless.
Make sure that you also include a method of measurement. With fake ads the channel takes care of this: Google, Facebook, and Twitter all give metrics on how your ads perform. And you might as well track the number of visits on the page you send them to. If the link is on your website, add a "Google tracking event" to your button, or create a special landing page so you can track the number of visitors.
Measure.
Run the experiment. And try to be patient. Don’t draw conclusions too soon; wait until the test is complete (in this case when you have at least 1000 visits or 10,000 views).
Interpreting Results
Check your initial hypothesis and see if the results match your success rate. Depending on your results, you have three options:
Success! Congratulations, you were right. You decide to build the feature.
No success! You pivot. Try reframing the link or button. A different word/icons/picture might yield a totally different outcome.
No success! Congratulations. You prevented yourself from wasting scarce resources.
Time Commitment
1 hour to formulate a hypothesis.
1 - 4 hours to create the fake doors, depending on the type.
1- 10 days to generate enough traffic for a representative sample size (depending on the amount of traffic or money you’re willing to spend)
1 hour to evaluate and write down learnings.
Potential Biases
Copy bias: The words used in the fake door will greatly influence your results. Make sure to try copy variations.
Location bias. The location of your ad (which platform) or your link (relative position on your website) will also influence your results. Make sure you try different variations.
Field Tips
Don’t scare off your users. Testing is okay, but be careful not to overdo it.
Making (too many) false promises by fake doors might give the user the impression of a lousy product or cause them to lose trust in your company.
Make sure you have a significant number of users in order to judge.
Got a tip? Add a tweetable quote by emailing us: realbook@kromatic.com
Case Studies
Game studio Zynga has a strong reputation for using fake doors and other quantitative product tests. They test new game ideas by coming up with a five-word pitch for each one. Then they publish the pitch as a promotional link in their live games for a period of time to see how much interest it generates from existing users.
Local Dutch information aggregator Tippiq.nl uses fake doors extensively in their proposition development using a separate website with www.tippiqlabs.nl. All new ideas are tested first using fake ads that lead to a landing page on that website. On the landing page the product/service is explained and the visitor is asked if he would like to use the product. After they have expressed their interest they get a notification that the product is still in development and they can leave their email if they want to stay up-to-date.
Got a case study? Add a link by emailing us: realbook@kromatic.com