Broken Promise Smoke Test

The Broken Promise Smoke Test is a great way to test the inherent virality of a product or startup idea

Jordan Duff avatar
Written by Jordan Duff
Updated over a week ago

Overview

Method category: Evaluative market experiment

How to Use This in GLIDR

The Broken Promise Smoke Test is a great way to test the inherent virality of an idea by seeing how many of your users refer someone to your product despite you telling them to keep it secret. 

In GLIDR, you should run an Experiment for this test with a hypothesis about the cutoff rate for how many referrals indicate a success. Then, add Evidence - Other with the data from your test once you've completed it. Finally, in the Analyze phase, assess how well this test worked and update your project accordingly.

Learn more about each of those aspects of GLIDR:

---

Broken Promise Smoke Test

Article excerpted from The Real Startup Book

In Brief

People naturally want to share ideas or products they love with their friends. This smoke test helps test how much of a natural/inherent virality lies in the idea itself. It helps identify high-potential ideas before you've built a product.

Helps Answer

  • Does the product answer an unmet need?

  • Are users getting excited enough to share the product or product idea?

  • Do you have an idea or product that people love?

  • What relationships exist in your market?

  • Who gets the most excited about the product?

Tags

  • Viral

  • Landing page

  • Sharing

  • Audience

  • Segmentation

Description

This smoke test puts up a small social barrier (the social pressure of a personal promise) to see if anyone would consider the product idea worthy of sharing regardless of how developed it is. It tests the virality coefficient, or the "Referral" in Pirate Metrics' AARRR model. It is arguably a social variation of the landing page MVP.

This test is most relevant for early stage founders, particularly when testing a variety of ideas. It helps identify which of a number of ideas actually have relative merit. Ultimately, it helps focus your resources on an attractive and buzzworthy product.

In terms of the mechanics, it works similar to a landing page. There is a dedicated web page with a call-to-action. On the thank you page or in follow-up communication, the copy explicitly asks the registrant NOT to share the idea with others. After some time passes, you compare the original list of people you contacted with the actual list of signups. Anyone who wasn't in your original list of people you told about about the idea is counted as a referral.

Referral rate = (total number of signups in a time period/total number of original people contacted)

If this is above 100 percent, then you are seeing referrals.

With this smoke test, you aren't testing conversion rate but referral rate. Has anyone referred the product despite being told that it's not to be shared?

To the extent possible, it's also worth noting who is being referred. This provides additional insight on the ideal target profile for that product idea. These referrals should ideally be interviewed to discover more about them.

Time Commitment

Minimal: 1-2 day developer and designer time, or you can use a viral landing page template like the ones at KickoffLabs.com

How To

  1. Send out a few early landing pages/code-free MVPs to a group of 10-15 people.

  2. Ask them not to share with anyone.

  3. Track email signups or a different CTA (follows, Facebook page likes).

  4. See which or any of the pages gets signups outside of the group you sent it to (broken promise = formula for growth).

Interpreting Results

Interpretation is framed by how you originally formulated the experiment, i.e., what is the cutoff value for the referral rate in order to make a go/no-go decision? Also, be clear on exactly who you count, both in the original and referred group. For example, immediate family like your mom doesn't count (as well as anyone she refers) unless she really is representative of your target market.

Potential Biases

  • Confirmation bias

Field Tips

  • “It’s only hard [to market an idea] if your idea or product isn’t loved.” @simonelucidi87

  • "Figure out your cutoff rate for referrals before you run a broken promise smoke test" @LaunchTomorrow

  • Got a tip? Add a tweetable quote by emailing us: realbook@kromatic.com

Case Studies

References

Did this answer your question?