Sample size calculators

Luckily, you are just seconds away from some very smart brand marketing solutions. Click here!

Key points

Broadly speaking:

• Define very clearly your target audience, and only include the people from whom you want the information. This may appear to be a silly point, but it is probably missed in over 75% of surveys
• Talk to as many people within that target audience as you can (afford). The more the better
• Beware of inequalities of buying power in business-to-business markets. One company may literally be worth 10,000 others

And above all, skip all the details that follow, and go right to the end where you will find two free sample size calculators.

Read the detail once you have played with the calculators.

In more detail…………..

The issue

Whenever one does market research, one of the key questions is “How many people should I talk to?”

Actually, before that, the questions should be:

1. whom do I want to talk to?
2. what information do I want at the end of it?

……..because the two most common criticisms of research are:

• you talked to the wrong people
• you asked them the wrong questions:
• because they did not understand the question
• because the results are not actionable

Only after that do people ask “Are the results significant?”

The simplest solutions

The simplest solutions are to say:

• for quantitative research (i.e. when you are adding up the numbers), that the more people you talk to within your target audience, the better the data
• for qualitative research (i.e. when you are trying to understand the range of possible answers), that you should carry on interviewing until people’s answers become predictable

Unless you are doing tracking research, it generally does not matter whether your results are likely to be +/- 5% or +/- 7%. The question is “Do you feel confident in the information?”

Qualitative research

For qualitative research, the numbers are usually relatively small (between 5 and 30 interviews), so you are outside normal statistical parameters, and anyway you are not counting. What you are doing is trying to understand:

• what the range of possible answers could be
• why those answers are as they are

Sometimes you can find two or three people who know everything (as against those who think they know everything), but it may take 20 interviews to find them.

Sometimes markets are very homogenous (everyone thinks the same thing), in which case 5 interviews are enough to map the territory. In other markets, views are very disparate, and it might take 50 or more interviews before you exhaust all the possible answers.

Business-to-business markets

In business-to-business markets there are usually massive inequalities of buying power. Two or three companies will buy in millions, and most of the rest will buy in thousands or in hundreds.

In this case you want to talk to all the Big Three (and analyse those separately), then talk to a sample of the middle range (and analyse those separately), then decide if you want to talk to the rest (you may, of course, find that this group is your most profitable one).

Consumer markets

Even in consumer markets you will find that 20% of your customers account for 80% of your profits, so you need to start here. Decide who are the consumers you really want to do business with, and then talk to a sample of them.

Validity and reliability

There are two other key concepts in defining your sample and your market research methodology:

1. validity – will the information you capture be reflective of reality? Some of that will be a judgment call
2. reliability – once you have decided that your results will be valid, will you reliably get the same results each time if you apply the same methodology?

Calculating sample sizes

Sample size statistics do not address the validity issue, except that the more people you interview, the more likely you are to avoid biased and skewed results.

They are fundamentally there to address reliability:

• are you getting a different answer from the last time because the situation is different, or simply because your methodology is not very reliable (it creates too much “noise”)?

There are different ways of addressing the issue of reliability. The simplest is to split your sample in half, and to analyse them separately. If you are getting more or less the same answer from both analyses, the chances are that your methodology is reliable.

So what is “more or less the same answer”?

There are two key concepts here:

1. Confidence level – how confident do you need to be that the two different surveys or groups are statistically different from each other. The usual market research confidence level is 95%. You want to be 95% confident that the two results are statistically different. For critical-to-life research, e.g. pharmaceutical testing, you will tend to choose 99%. The higher the confidence level, the less likely it is that you will find a statistical difference
2. Margin of error/confidence interval – according to the confidence level you choose, and according to where the results fall within a range (i.e. 50% vs. 5% or 95%), the +/- margin of error tells you where the real world answer is likely to be (assuming that your methodology is valid in the first place). So, if you get an answer of 55%, with a +/- of 10%, the true answer is likely to be between 45% and 65%. This is why tracking surveys require much larger sample sizes. If the only way that your results can be statistically different is that they are 45% one week and 65% the next, the chances are that there is something seriously wrong with your survey methodology. The world does not usually work that way

There is one further complication: you do not calculate statistical significance by simply adding together the margins of error for the two surveys, and seeing whether the difference between the results exceeds the combined margin of error. It is very unlikely that both survey results will be on the extremes of the possible answers. There is therefore an additional piece built into the formula to take this into account.

Calculators

All statistical packages will calculate the margin of error of an answer at a given confidence level, but you will have to have all the raw data in there first (i.e. what each respondent said).

However, there are two packages around that will calculate the margin of error of a sample from the result – i.e. “50% said this week”.

Mud Valley™ Better-than-guessing™ calculator

The Mud Valley™ Better-than-guessing™ calculator was designed to calculate whether the answers from two separate surveys are statistically different or not. You put in your sample size for each survey, and you enter the answer to the question in each survey (50%, 75%, 95% etc.), and the calculator tells you whether the results are statistically different at three levels of confidence – 68%, 95% and 99%.

It will also show how close they are to being statistically different (as far as we know, this bit is unique).

It costs £9.99 (all inclusive price) via our shop.

The Creative Research Systems calculator

CRS’ The Survey System calculator will not tell you whether the results from two different surveys are significantly different, but it will tell you the margin error for any given survey result.

They have also included an extra gizmo for where you have very small markets. Type in the total population size, and the calculator will reduce the number of people you have to talk to.

Of course, if you have a very small total market universe, the chances are that it is a business-to-business market with major inequalities of purchasing power, so be careful how you use this – but it is a neat gizmo nonetheless.

The site also contains lots of extra information on survey design that you might find helpful.

Click here to access the site.

Luckily, you are just seconds away from some very smart brand marketing solutions. Click here!

For further information, please contact enquiries@mudvalley.co.uk

© 2004, Mud Valley ™ brand marketing community.

Related answers