As stated earlier in this section, statistical significance – determined by the p-value – is not the same as effect size or practical/clinical importance (i.e. whether it “matters”).

Example A pharmaceutical company has developed a drug to cure obesity. During tests of this drug, it appears as migraine could be one of the side effects of taking this drug. The null hypothesis would here be that there are no differences in the risk of migraine between people who had taken the drug and those who have not. The alternative hypothesis would then be that there are differences. When we run the analysis on this data material, we see that those who have taken the drug have ten times the risk of migraine, but the p-value is above the 95% confidence level (i.e. p>0.05). Thus, we cannot reject the null hypothesis. The difference is however large and is likely to have significant impact on people’s lives. It could moreover be the case that a type II error has occurred here due to a small sample size.

Example Researchers have gathered data on coffee consumption and happiness among 100,000 company employees. The null hypothesis would here be that there are no differences in happiness between people who drink coffee and those who do not. The alternative hypothesis would be that there are differences. The analysis suggests that there is a tiny difference in happiness between those who drink coffee and those who do not, to the advantage of the coffee drinkers. The p-value is below 0.05 which suggests that the null hypothesis can be rejected at the 95% confidence level. However, the difference is very small and the results may not be very useful.