As an independent third party that assists associations in their research efforts, Vault is often asked if the results of research projects are “significant”. Clients collecting data ask this question in reference to the term statistical significance, but are often seeking to understand if the data they’ve collected is useful or meaningful and whether or not it benefits their future marketing or membership efforts.
Here, we want to help clarify what statistical significance means and when it applies to research. Then, we want to identify ways to make sure that the data you collect will set you up for success as you further the research efforts within your organization.
Statistical significance is a measure used when surveying a representative sample of a population in order to determine if there is a relationship between data points. Most basically, statistical significance reveals whether or not the results from the research are due to chance. Let’s think of it this way – if we ran the same study 100 times, can we expect the outcome to show similar results each time, thus eliminating the element of chance or sampling error in the results?
There are several terms that you will need to be familiar with when using statistical significance:
Statistical significance reveals a relationship amongst specific data points within the overall results. A statistical significance analysis is often used to compare two data points or focus in on one sample to view any changes over time. The relationships uncovered from the statistical significance analysis can help draw conclusions about specific findings and create a narrative about the research as a whole.
If you’re interested in knowing how to calculate statistical significance, there are many programs and online calculators that can do the math for you. Many statistical packages come with a feature to highlight data that is significant. Crosstabs are also a useful tool in both dissecting the data as well as viewing statistical significance. Again, statistical packages often provide the ability to present the data in a crosstab format, which allows options to view the data by different sorts or questions.
Although a valuable tool in the world of research, statistical significance’s usefulness depends on the type of research you initiate. In research, you can either survey a sample from the whole population, or perform an audit, or census, in which you survey the whole of your membership, industry, etc.
When utilizing a sample, researchers select a random subset that is representative of the larger population. The sample is either selected from within the membership or obtained through an independent list. The goal of this type of research is to ask questions to this smaller group in order to gain an understanding of the ideas and preferences of the overall population. Results collected from the sample allow a story to be told about the population as a whole. We apply the confidence level to determine the degree to which we can assume the results of the sample depict the actual results of the entire population.
Alternatively, an audit collects existing data from individuals that span an entire population, for example an entire association membership or industry. When including an entire group in the research, you will collect the same information from all individuals. The data can focus on topics, such as membership satisfaction or the amount of product sold in a quarter. Since this type of study reaches all individuals, the only potential for missing data would be gaps in data due to non-response or human error. Non-response bias, which occurs due to the difference in the actual results compared to what the results would show if everyone participated, is an issue facing all researchers and often a topic we discuss with our clients. Unfortunately, you cannot project the results found in the audit to cover the missing pieces where participants have chosen not to or are unable to complete all or part of the survey. However, in the section below, we will cover ways to help maximize participation and minimize common errors.
As a research tool, statistical significance uses findings from the sample to project results to a population; therefore, it is only relevant to the sample type of research explained above. When performing an audit/census, you are reaching the entire population and do not need to project your results to a larger group and, therefore, will not need to use statistical significance with the results.
Whether or not the type of research you implement utilizes statistical significance, the research for your organization and from your members is always important and useful!
Regardless of the type of research, keep the following ideas in mind in order to ensure you get the most out of your data: