You are convinced that you have the worst luck when it comes to cars. You've had your car towed, stolen, totaled, totaled again, and you always get a parking ticket even if you're only 2 minutes late. You want to know if this is all just due to chance or if there may be something else going on. These are the same questions research psychologists ask when conducting a study: Is it by chance or by some other factor? Enter statistical significance.
What is the definition of statistical significance?
How is statistical significance determined?
What formula is used to find statistical significance?
What is an example of statistical significance?
How is statistical significance used in psychology?
Statistical Significance Definition
One of the most common ways researchers attempt to answer a question is by comparing two samples and seeing if there is an observed difference.
Observed Difference: refers to the way that two groups are unlike one another.
Depending on several factors, this observed difference can be either due to chance or some other significant factor. But how do we know the difference? The best way is to determine whether the observed difference is statistically significant.
Statistical Significance: a term used by research psychologists to understand if the difference between groups is because of chance or if the difference is likely because of experimental influences.
Researchers are especially interested in statistical significance during hypothesis testing. Two types of hypotheses are considered in hypothesis testing: the null hypothesis (H0) and the alternate hypothesis (H1).
Null Hypothesis (H0): states that the observed difference between sample groups is due to chance.
Alternate Hypothesis (H1): states that the observed difference between sample groups is not due to chance but some other factor.
If an observed difference is found to be statistically significant, we can reject the null hypothesis and accept the alternate hypothesis.
Fig. 1, What are the odds, Pexels.com
Determining Statistical Significance
Determining statistical significance should first begin with finding the effect size.
Effect Size: the size of the observed difference found between groups.
Two essential things must be true about the samples taken.
The sample must reliably represent the population, meaning there should be low variability within the group.
The sample size must be large enough. It may be a less accurate representation of the population if it's too small.
Once the effect size is determined, we can find the value that will tell us if the effect size was just a fluke or due to some other factor. This value is called the p-value.
P-Value: the probability that, if we were to repeat a study several times, we would get an observed difference at least as extreme as our actual sample, given the null hypothesis is true (it's by chance).
If this number is below the significance level or the value set at the start of the study, we can reject the null hypothesis, meaning the results we obtained were not due to chance.
Statistical Significance Formula
In order to find the statistical significance of a study, we must find the p-value. This can be complicated, so we use several different tables that do the hard part for us. However, to read these charts, there are a few things we need to understand first.
Earlier we mentioned that for the effect size to be reliable, the sample must be from a large sample and have low variability. When these two things are true it should create a curve with a normal distribution.
Normal distribution curve: a symmetrical curve that displays a continuous probability distribution.
Fig. 2, Normal distribution curve displays a continuous probability distribution, Commons.Wikimedia.org
The next thing we need to understand for the statistical significance formula is a test statistic. Many times, researchers will find the z-test statistic. The z-test statistic essentially takes the data we collected including the sample mean, sample standard deviation, and sample value, and gives us one single value. The type of test we perform tells us which tail end of the curve we pay attention to -- lower-tailed, upper-tailed, or two-tailed test.
Fig. 3, Upper-tailed test, Commons.Wikimedia.org
Now, let's put everything together to find our p-value. Once we've found our z-test statistic, we find the point on our normal distribution curve. If it is an upper-tailed test, we are paying attention to the area to the right of the z-test statistic. The value of this area is the p-value. As we mentioned earlier, while there is a formula to find this area, it's a bit complicated. So instead, we use p-value charts or calculators to find our value.
Statistical Significance Psychology
Statistical significance in psychology can be an important value to know. Psychologists study the mind and behavior. While psychology is a science, the mind and behavior can be difficult to measure.
If we observe a difference in how many times a car runs a red light at one intersection versus another, how do we know this observation wasn't just a coincidence? What if we just picked days when there was more traffic at one intersection than the other? Finding the p-value will help us answer this question.
Psychologists are very cautious when it comes to statistical significance. They may set the significance level at 0.05 or even as low as 0.0001 which would increase the study's significance. Psychologists want to be confident that their result was not a fluke. And even still, the study may not have any real meaning if the effect size is extremely small. Even if a difference is not likely due to chance, it might not be a very significant difference at all.
Psychologists want to know how they can apply the results of a study to the real world. Just because we reject the null hypothesis, does not mean it will have any kind of effect outside of the lab.
Finally, it's important to note that even if you get a p-value above your significance level, it does not mean that your result is definitely because of some random event. It just means you can't be too confident that it is not. Statistical significance simply gives psychologists more information to help them ask or answer more questions.
Statistical significance can help psychologists decide if a type of mental health treatment is effective or not. This can help determine which practices to stop and which to keep exploring.
Statistical Significance Example
Let's set up a hypothesis test as a statistical significance example. Say you want to see how many students go to college at your school compared to the national average. Here are your hypotheses:
Null hypothesis: the observed difference between your school and the national average is due to chance.
Alternate hypothesis: the observed difference between your school and the national average is due to something other than chance.
You set our significance level at 0.01 which means our probability that the observed difference is due to chance must be less than 0.01 before you can reject the null hypothesis. You get a z-test statistic of -2.43 and a p-value of 0.0075. This value is less than your significance level, therefore, your results are statistically significant and the null hypothesis can be rejected.
Statistical Significance - Key takeaways
- Statistical Significance is a term used by research psychologists to understand if the difference between groups is because of chance or if the difference is likely because of experimental influences.
- The sample must reliably represent the population it represents meaning there should be low variability within the group. The sample size must be large enough. If it's too small, it may be a less accurate representation of the population.
The statistical significance formula is based on a normal distribution curve. The p-value is the area between the z-test statistic and the tail end of the curve (depending on the type of testing).
Psychologists are very cautious when it comes to statistical significance. They want to be confident that their result was not likely due to chance.
Even a study that is statistically significant may not have any real meaning if the effect size is extremely small.
References
- Fig. 3 - Bell Curve (https://commons.wikimedia.org/wiki/File:BELL_CURVE.png) by Lawrence Seminario Romero is licensed by CC BY-SA 4.0
Explanations 
Exams 
Magazine 
