Most founder surveys are written to confirm something the founder already believes, and they succeed. The result feels like evidence, gets put in a deck, and a decision worth six figures gets made on a number that was decided before anybody was asked anything.
How do you run an unbiased survey?
Three things have to be right and most surveys get one. Ask people who represent the market rather than people who already like you. Write questions that do not contain their own answer. And decide what you will do with each possible result before you see any of them.
That third one is the cheapest and the most skipped. Write down, in advance, what number would make you abandon the idea. If no number would, you are not running research, you are commissioning a quote for a slide, and there are faster ways to get one of those.
What makes a survey question biased?
A question is biased when its wording, its options or its position makes one answer easier to give than another. The three common forms are leading wording, unbalanced scales, and order effects, and all three are invisible to the person who wrote them.
| Problem | What gets written | What to write instead |
|---|---|---|
| Leading wording | "How useful would an automatic weekly report be?" | "How do you currently produce your weekly report?" |
| Unbalanced scale | Excellent, very good, good, fair, poor. | Equal counts either side of a neutral midpoint. |
| Assumed premise | "What frustrates you most about your current tool?" | "Do you use a tool for this? If so, what happens when it goes wrong?" |
| Order effect | Price asked after five paragraphs on the benefits. | Price asked first, or on a randomised page order. |
The left-hand column is not incompetence. It is what anybody writes about something they have been thinking about for a year, which is exactly why the person who wrote the product should not be the only person who writes the questions.
Who should you survey?
People who match the market you intend to sell to, recruited from a panel rather than from your own audience. Your followers, your mailing list and your Twitter replies are the most enthusiastic slice of the people most likely to already agree with you, which makes them the worst available sample for a question about demand.
The cost of getting this wrong is specific and large. A survey of your own audience will tell you a product is wanted, because the people who follow you follow you for a reason. The market is mostly people who have never heard of you and have a functioning alternative, and they are the ones who decide whether the business works.
- Your own audience
- Representative panel
Illustrative, not a study. The shape is what matters and it shows up almost every time: an owned audience rates everything high and rates them in a different order. If you had chosen from the first line you would have built Concept A, and the market preferred B.
Growth is a process. Nothing here happens overnight, and anybody promising you overnight results is lying to you.
Ask the market from $100How many responses do you need?
For a single national figure you want a few hundred from a representative sample, which gives a margin of error around five points. For comparing groups you need that many in each group. Small samples are not useless, but a difference of six points on eighty responses is not a difference.
The number people fixate on is the response count, and the thing that actually determines whether the result means anything is who those responses came from. Two thousand responses from a self-selected audience is a worse instrument than three hundred from a representative panel, and it is more convincing, which is what makes it dangerous.
Can you use a survey to set your price?
You can use it to rule prices out, not to pick one. What people say they would pay is reliably higher than what they do pay, and the gap varies by category. The useful output is the shape of the curve: where interest falls off a cliff, and where it was already gone.
Two techniques survive contact with reality better than asking directly. Ask what they currently spend on the problem, including the hours, because that is a number about the past rather than an intention about the future. And put the price in front of a concept before describing the benefits, so you are measuring a reaction rather than a negotiation.
What should a research deliverable include?
The raw responses, the exact question wording, the sampling frame and the methodology, alongside the findings. Anyone who gives you conclusions without those four is asking you to trust the interpretation without being able to check it.
This matters more than it sounds, because the same dataset supports several readings and the one you are shown is the one somebody chose. If the working is attached, a sceptical person on your own team can check it, and a finding that survives that is worth acting on. A finding that cannot be checked is a slide.