- How different? How similar?
In research the main function of statistics is two ask 2 questions:
- How different? - Are the results from this year really different from the results from last year? Is this group of people really different from that group of people?
- How similar? - Which of these issues belong together? Which of these brand values correlate with brand loyalty?
- Statistics as an insurance policy
The main point about statistics is that they act like an insurance policy. They ask "Can I be sure that these results are really different (or really the same)?" - and their approach is essentially conservative.
- Types of statistics
There are also two different types of statistics:
- normal distribution statistics - which are the most common type, and are used on larger surveys
- non-parametric statistics - for use with surveys of less than 30 people (these will not be covered here)
- Normal distribution statistics
Normal distribution statistics are based on the discovery that responses, or "observations", in the world seem to work to a common pattern - a "bell curve" shape. The majority of people (68%) seem to cluster within 1 interval ("standard deviation") either side of the middle, 95% within 2 intervals, & 99.9% within 3 intervals.
- Margins of error
When you conduct a piece of marketing research, unless you take a "census" (i.e. interview everyone), you interview a sample - a smaller group of people designed to be representative of everyone. While the spread of results should have the same bell curve shape as if you had interviewed everyone (unless the sample is not representative of everyone), it will probably not be in quite the same place - there will be a "margin of error". The question then is "How far are the results from the survey likely to be from the results you would have got if you had interviewed everyone?"
- Confidence and distance
This is a question of:
- Confidence - how confident can you be?
- Distance - how far "off-centre" are the results likely to be?
The larger the sample you interview, the more confident you can be as to the distance between the results you received from the survey, & the results you would have received if you had interviewed everyone.
- Confidence - "I can be 95% confident"
- Distance - "that real results are within + or - 5% of my results"
- Extremes and the middle of the scale
If a very high or very low percentage of your sample say they are aware of your brand, you can be more confident of the result than if only 50% say it. 50% awareness will have a higher margin of error for the same level of confidence than 90%, or 10%, awareness.
- 90%, 95% & 99% confidence
For clinical trials in the pharmaceuticals industry, you want to be 99% certain that the drug works and has no identifiable side effects. This may extend to other products that can affect your health, such as detergents or foods.
For general marketing purposes, 95% confidence has been the standard threshold of confidence.
However, even 95% confidence requires relatively large sample sizes to determine the difference between one year's results and the next, so many research agencies have reduced the threshold to 90% confidence.
- No statistical difference does not mean no difference
If you get a higher awareness in the second survey than in the first, but it is not statistically significantly higher, it does not mean that your awareness has not improved, only that you cannot be sure it has improved.
- "Top Box" analysis
A few years ago, someone spotted that people who score you 5 out of 5 for satisfaction with your brand are very much more likely to continue to buy it than those who score you 4 out of 5 (six times more likely, in fact). This has led to the "Top Box" analysis where you only count the highest point(s) on the scale - 5 out of 5, 9 or 10 out of 10.
This approach delivers a highly discriminating analysis, especially between different brand attributes, where some may score 80% Top Box, and other 15%. It also works well on overall loyalty scores, reducing 90% very/quite satisfied ("Not much to do, then"), to 30% very satisfied ("Lots to do").
- Cultural differences
There are, however, cultural differences in the way people score you. French people, for instance, are very reluctant to score you 5 out of 5. Germans tend to prefer scales where 1 is the highest score.
Some of the most sophisticated agencies have methodologies for identifying on an individual level what is a high score for that individual by looking at the pattern of that person's responses throughout the survey.
- Skew and kurtosis
You may hear the expressions "skew" & "kurtosis":
- a "skewed sample" is where the shape of the responses is off-centre -weighted to one end of the scale or the other
- a "kurtosis" is where the results are bunched more tightly around the mean than is usual - with a tall peak in the middle
Both results mean that your sample or methodology is selective in some way - e.g. you may have interviewed only your customers (skew), or used a limited scale (kurtosis).
- Correlation
The other major issue in branding research is correlation - how similar is one result to another? If your "friendliness" scores rise, do your "intimacy" scores rise too, & in what proportion?
There are 2 types of correlation:
- positive correlation - as one score increases, the other score increases
- negative correlation - as one score increases, the other decreases
The maximum positive correlation is +1. The maximum negative correlation is -1. So all correlation scores run between +1 & -1. 0 means there is no correlation at all between the two results.
You then get levels of confidence in much the same way as before - you can be 99% confident, 95% confident, 90% confident, or less. The more extreme the correlation (towards either +1 or -1), the more confident you can be.
- Correlation is not causation
The thing to remember about correlation is that high correlation does not mean causation. If it is very hot, you may take your shoes off, & you may eat an ice-cream - it does not mean the ice-cream makes you take your shoes off, or that taking your shoes off makes you eat an ice-cream.
- Categorical statistics
There are also statistics used to identify statistically significant differences in categories as against numerical scores. For instance, if you have 2 groups who have scored you on a scale between very satisfied and very unsatisfied, are these 2 groups statistically significantly different as groups, or are they following a similar pattern?
Categorical statistical testing is normally conducted using Chi2 techniques. These work by estimating the number they expect to see in each "cell" (e.g. 25%, 50% etc.), & how different the number actually is from what was expected. Again, you can have different levels of confidence in the answer.
- Cluster and factor analyses
Cluster and factor analyses are used extensively in brand research to identify market segments and key attributes.
"Cluster" analyses examine whether different people tend to answer the same way. If so, they may be a market segment. Correlation and Chaid (a derivative from Chi2) are the most common techniques for defining clusters, but pattern matching techniques can also be used.
"Factor" analyses examine whether different questions are answered the same way. If so, they may be related to the same higher level issue. Factor analysis is a way of reducing 40 detailed attributes down to 5-6 drivers of customer behavior.
- Predictive techniques
Chaid is also used to predict people's responses, especially to marketing programs.
A newer type of predictive analysis uses "neural networks". This is a pattern matching technique, based on the observed behavior of the brain, that uses repeated iterations of inputs & outputs to predict future outcomes.
Unlike most other types of statistics, neural networks are not told what to do, they are "trained". At each iteration of the input-output cycle they try to estimate the next output from the input data. As time goes on, they will become increasingly accurate at predicting outcomes, assuming you provide them with the right input date.
Neural networks are mostly used in forecasting - sales, events etc.. In tests they have often proven more accurate than more established chaid statistical techniques at estimating levels of response to, for instance, mailshots.
- Conjoint analysis
Conjoint analysis is a statistical technique that allocates a "utility value" to elements of a package - for instance, the price of a product or the component of a car.
Respondents are offered two or more options, each comprising several features (size, weight, price, speed, color etc.). They choose one option. This is repeated many times until a sufficient number of permutations of the features have been completed.
Conjoint analysis then allocates a utility value to each feature and thus defines how big an influence each feature had on the choices that were made.
- Perceptual maps
The standard ways of presenting statistics are in tables or graphs (normally pie charts or bar charts). In branding, perceptual maps are also used. These are two-by-two matrixes.
There are two types of perceptual map:
- ones where the axes are labelled (e.g. importance vs. performance)
- ones where the axes are not labelled
Where the axes of the matrix are labelled, it is simply a question of where the attribute or brand fits within the matrix.
Where they are not labelled, the aim is to show correlation - how close different attributes and brands are to each other. Those that are drawn closest together "occupy the same space" (whatever that may mean).
- Why you should probably not have started reading this
If you feel you are getting to hung up over statistical significance, remember what can go wrong in research:
- you can interview the wrong people
- you can ask them the wrong questions
- you can phrase the questions in a confusing way
- the interviewee can misunderstand the question
- the interviewer's attitude/dress may encourage the interviewee to give a misleading answer
- the interviewer can misunderstand the answer
- the interviewer can write down the wrong answer (accidentally or deliberately) or put it in the wrong place on the questionnaire
- the person inputting the data onto the computer may misread the answer or hit the wrong key or the data may become corrupted
- the analyst may come to the wrong conclusion based on the computer analysis
- the marketer may misunderstand the analyst
Are you still worried about the finer points of statistical significance?
