There are about 250 million adults in America, of every imaginable background and circumstance. So how can a survey of only 800 or 1,000 adults reflect what the entire country is thinking? How can a thousand voices speak for them all?
Marketing researchers liken it to making a big pot of soup — to taste-test the soup, you don't have to eat the whole pot, or even a whole bowl's worth. You only have to try a spoonful or two. The same is true of marketing research. You don't have to ask every single person in America to find out what Americans think; you only need to ask a few to get the flavor of the population’s opinion.
This fact is reflected by a survey's standard error of the mean. Specifically, the standard error of the mean is an index of the amount of error that results when a single sample mean is used to estimate the population mean, it is an index of sampling error. The standard error of the mean equals the standard deviation of the population of raw scores divided by the square root of the size of the sample on which the means are based.
So, a subject highly related to sampling error and standard error is the “margin of error.” The lower the margin of error, the larger the sample, and the more accurately the views of those surveyed match those of the entire population.
When marketing researchers report the margin of error for their surveys (usually expressed as something like "plus or minus 3 percent") they are stating their confidence in the data they have collected.
You must also remember that every margin of error has a "confidence interval," usually 95 percent. That means that if you asked a question from this survey 100 different times, 95 of those times the results would be within 3 percentage points of the original answer. Of course, this means that the other five times you ask the question, you may get answers that are completely different.
For example, if 50% of a sample of 1,000 randomly selected Americans said they are satisfied with their bank, in 95 cases out of 100, 50% of the entire population in the U.S. would also have given the same response had they been asked, give or take 3 percentage points (i.e., the true proportion is somewhere between 47% or 53%).
The bigger the sample, the smaller the margin of error, but once you get past a certain point -- say, a sample size of 800 or 1,000 -- the improvement is very small. The results of a survey of 300 people will likely be correct within 6 percentage points, while a survey of 1,000 will be correct within 3 percentage points, a lower margin of error. But that is where the dramatic differences end -- when a sample is increased to 2,000 respondents, the margin of error drops only slightly, to 2 percentage points.
Despite this, some surveys have sample sizes much larger than 1,000 people. But why ask two or three thousand respondents when 800 will do? Well, it sounds more impressive, but that's hardly worth the cost of interviewing all those additional people. Usually when a study has a large sample, it is so certain subgroups can be isolated and compared to other subgroups or to the total sample. If you want to compare retired people to the general public, for instance, a sample of 1,000 might yield only one or two hundred people who are no longer working, which may not be enough to get a solid grasp on the views of that group. A sample of 2,000, however, will probably yield a larger group of retired Americans and provide a more accurate picture of their views, which can be compared to non-retirees’ opinions.
Sometimes increasing the overall sample size is not enough, if the subgroup you are examining is rare or particularly hard to find. Affluent households, for example, make up only a small percentage of the U.S. population. In a standard random sample, you would have to interview an enormous number of people before you had a large enough subgroup of affluent households. In this instance, you would take an "oversample," purposely seeking out members of the “high-net worth” group you are interested in, and comparing the results to the main sample.
Of course, in both general samples and oversamples, who is asked is as important as how many are asked. Reputable survey organizations go to great lengths to make sure their interview sample is random and representative of whomever they are surveying, be it retired, affluent, or all Americans.
Sometimes, even the best researchers misuse and abuse the concept of significance. Many in research pour over reams of cross-tabulations and perform a multitude of analyses to find significant differences and formulate their decisions based on statistical significance. They tend to associate statistical significance with the magnitude of the result. Their reasoning is something like this: “The more statistically significant a result, the bigger the difference between two numbers.” In other words, the fact that one proportion is significantly different than another suggests that there is a big difference between the two proportions and statistical significance is often associated with “bigness” of a result.
People often think that if the difference between two numbers is significant it must be large and therefore must be considered in the analysis. It is suggested that when comparing numbers, two types of significance should be considered: statistical significance and practical significance. By understanding the difference between statistical and practical significance, we can avoid the pitfall that many in the research industry make.
What does statistical significance mean? A significance level of, say, 95% merely implies that there is a 5% chance of accepting something as being true based on the sample when in the population it might be false. The statistical significance of an observed difference depends on two main factors: the sample size and the magnitude of the difference observed in the samples.
For example, let’s say we do a significance test between two groups of people who are exposed to a product concept and find a 20-point difference between Group A (65% acceptance) and Group B (45%). Is the difference statistically significant? Despite the large magnitude of the difference (20 points), its statistical significance will depend on the sample size. According to statistical theory, we need a sample size of about 50 or more people in each of the groups for the difference to be statistically significant at the 95% level of confidence. If we meet the sample size requirement, then the difference of 20 points will be statistically significant at the 95% level of confidence.
What does this really mean? Many marketers will look at this result and conclude that since there is a 20-point difference and the difference is statistically significant, there must be a big difference between Groups A and B. In reality, if we had done a census (i.e., surveyed the entire population) instead of surveying a sample, the difference between Group A and Group B may have turned out to be smaller. In other words, what this result tells us is merely this: given our particular sample size, there is a 5% chance that in the population represented by this sample, the proportions for Group A and Group B are not different. That’s all!
Statistical significance does not tell us anything about how big the difference is. It only tells us the probability with which a difference found in the sample would not be found in the population. Thus, for this case, statistical significance would allow us to conclude that there is only a 5% chance that in the population the proportion of Group A favoring the product is not higher than Group B; we are taking a 5% risk of concluding a difference exists when there may not be any such difference. If this difference were significant at the 99% level of confidence, it would not have become larger. It would only mean that there is a 1% chance that the difference observed in the sample would not be observed in the population. Thus, we are only taking a 1% risk.
From a marketing perspective, the statistically significant difference of 20 points may be meaningful or meaningless. It all depends on our research objectives and resources. If it costs millions of dollars to reach each additional percentage of the market, we may decide to funnel resources toward Group A since it has a higher acceptance rate. In this case, the difference may be termed a “big” difference because (a) we are reasonably sure (95% or 99% sure) that the difference observed in our sample also exists in the population and (b) each percentage of difference is worth millions of dollars to the client. Thus, statistical significance should not be used to decide how big a difference is but merely to ascertain our confidence in generalizing the results from our sample to the population.
In another situation, the same difference may be ignored despite the fact that it may be statistically significant. For instance, if the marketing costs are so low that it makes sense to market to both groups, we can ignore the difference (even though it is significant) and treat both groups as if they were the same. We may choose to market to both groups as if they had similar acceptance rates (even though our statistical test was significant).
The logic is the following: although we can be 95% sure that the difference observed exists in the population, given the marketing scenario, the difference is not meaningful. Thus, the relevance of a statistically significant difference should be determined based on practical criteria including the absolute value of the difference, marketing objectives, strategy, and so forth. The mere presence of a statistical significance does not necessarily imply that the difference is large or that it is of noteworthy importance.