Low power reduces your chances of discovering real findings. However, many analysts don’t realize that low power also tends to exaggerate the effect size when they detect effects.

there is a tradeoff between Type I and Type II errors. If you hold everything else constant, as you reduce the chance for a false positive, you increase the opportunity for a false negative.

(significance level). Your sample evidence provides sufficient evidence to conclude that the effect exists in the population.

The significance (alpha) level—how far out from the null value is the critical region?

On the probability distribution plot, the significance level defines how far the sample value must be from the null value before we can reject the null hypothesis.

If you think back to the sampling distributions, it makes sense. The sampling distributions assume the null hypothesis is correct. The significance level defines the critical regions. Therefore, when the null hypothesis is right, you expect test results to fall in the critical regions with a probability set by the significance level.

Even though we don’t know which studies have false-positive results, we do know their rate of occurrence. The rate of occurrence for Type I errors equals the significance level of the hypothesis test, also known as alpha (α).

Statistical power is the opposite of Type II errors, both mathematically (1 – β) and conceptually. Power is the ability of the test to detect an effect that exists in the population. In other words, the test correctly rejects a false null hypothesis.

hypothesis tests make assumptions about the data collection process. For instance, these tests assume that the data were collected using a method that tends to produce representative samples.