Is Your Data “Significant”?


As an independent third party that assists associations in their research efforts, Vault is often asked if the results of research projects are “significant”. Clients collecting data ask this question in reference to the term statistical significance, but are often seeking to understand if the data they’ve collected is useful or meaningful and whether or not it benefits their future marketing or membership efforts.

Here, we want to help clarify what statistical significance means and when it applies to research. Then, we want to identify ways to make sure that the data you collect will set you up for success as you further the research efforts within your organization.


Statistical significance is a measure used when surveying a representative sample of a population in order to determine if there is a relationship between data points. Most basically, statistical significance reveals whether or not the results from the research are due to chance. Let’s think of it this way – if we ran the same study 100 times, can we expect the outcome to show similar results each time, thus eliminating the element of chance or sampling error in the results?

There are several terms that you will need to be familiar with when using statistical significance:

  • The null hypothesis states there is no difference between your data points. This is what you are testing, or the question which statistical significance is looking to answer. If you find statistically significant results, you will reject this null hypothesis.
  • The sample size refers to the number of people included in your research sample.
  • The confidence level defines what amount of error you will accept in your results. If you apply a 95% confidence interval, that’s similar to saying, “I am confident that if I ran this test 100 times, then my results would be similar to what I’ve found, at least 95% of the time.” A 95% confidence interval applies a 5% margin of error, which means there is a 5% probability that the results found are due to chance. The 95% confidence interval is standard for most research, shifting to a higher percentage (i.e. 99%) in research where safety is concerned – medical research, structural research, etc.

Statistical significance reveals a relationship amongst specific data points within the overall results. A statistical significance analysis is often used to compare two data points or focus in on one sample to view any changes over time. The relationships uncovered from the statistical significance analysis can help draw conclusions about specific findings and create a narrative about the research as a whole.

If you’re interested in knowing how to calculate statistical significance, there are many programs and online calculators that can do the math for you. Many statistical packages come with a feature to highlight data that is significant. Crosstabs are also a useful tool in both dissecting the data as well as viewing statistical significance. Again, statistical packages often provide the ability to present the data in a crosstab format, which allows options to view the data by different sorts or questions.


Although a valuable tool in the world of research, statistical significance’s usefulness depends on the type of research you initiate. In research, you can either survey a sample from the whole population, or perform an audit, or census, in which you survey the whole of your membership, industry, etc.

When utilizing a sample, researchers select a random subset that is representative of the larger population. The sample is either selected from within the membership or obtained through an independent list. The goal of this type of research is to ask questions to this smaller group in order to gain an understanding of the ideas and preferences of the overall population. Results collected from the sample allow a story to be told about the population as a whole. We apply the confidence level to determine the degree to which we can assume the results of the sample depict the actual results of the entire population.

Alternatively, an audit collects existing data from individuals that span an entire population, for example an entire association membership or industry. When including an entire group in the research, you will collect the same information from all individuals. The data can focus on topics, such as membership satisfaction or the amount of product sold in a quarter. Since this type of study reaches all individuals, the only potential for missing data would be gaps in data due to non-response or human error. Non-response bias, which occurs due to the difference in the actual results compared to what the results would show if everyone participated, is an issue facing all researchers and often a topic we discuss with our clients. Unfortunately, you cannot project the results found in the audit to cover the missing pieces where participants have chosen not to or are unable to complete all or part of the survey. However, in the section below, we will cover ways to help maximize participation and minimize common errors.

As a research tool, statistical significance uses findings from the sample to project results to a population; therefore, it is only relevant to the sample type of research explained above. When performing an audit/census, you are reaching the entire population and do not need to project your results to a larger group and, therefore, will not need to use statistical significance with the results.

Whether or not the type of research you implement utilizes statistical significance, the research for your organization and from your members is always important and useful! 


Regardless of the type of research, keep the following ideas in mind in order to ensure you get the most out of your data:

  • PURPOSE: The first step in any research effort should be to define the purpose of your project. Make sure every question relates back to this purpose and any reports or presentations highlight how the results reflect this purpose.
  • AWARENESS: If your research requires involvement from your membership, build awareness of your research before and during the process by sending emails and notifications to members. Upfront transparency with participants should communicate the purpose of the research and expectations in terms of time and type of data collection.
  • CONFIDENTIALITY: Most data collected is highly sensitive; therefore, ensure confidentiality with your participants. People are often reluctant to share information vital to their company and rightfully so. Again, share the purpose of the data up front and also explain how the results will be used and disseminated. Report all data in an aggregate format, following safe harbor laws so no company info will be compromised. Consider offering to sign an non-disclosure agreement (NDA) with participants if needed. Finally, outsourcing the data collection and analysis to an independent third party can further emphasize the importance of confidentiality.
  • TIMING: Give participants enough time to complete a survey before the final deadline and ensure all questions are clearly written so participants understand what information to provide. People are busy and often inundated with emails and requests. Understand participants’ time is valuable and do the front work in assuring your survey is error free, and easy to complete.
  • BREVITY: Make sure every question asked is necessary; eliminate questions that do not relate directly to the purpose of the research as unnecessary questions can hinder participation. Issuing clean, concise and purposeful surveys will give your membership confidence in future surveys you deploy.
  • INCENTIVE: Offering incentives – free registrations to a conference, gift cards, money, and sweepstakes – often boosts participation rates. Think about what incentives will relate well to your participants.
  • ACCURACY IN REPORTING: All of your hard work on building and executing the research culminates with a final report. This is what you will share with others, so it is vital that you take the time to thoroughly review the data results in order to produce reliable reports that are free of any typos or data errors.
  • ANALYZING YOUR INDUSTRY: Are you seeking to gain additional insight into your industry that previous research has been unable to capture? Or, is there a section of your market that you’ve been unable to reach? Think about implementing market research tools, such as in depth interviews or focus groups, to collect additional data from individuals related to your industry and dig deeper into the issues your membership faces.