What is Test of Significance: Methodology and Types

Tests of significance indicate the probability that the outcome of a study has occurred entirely due to chance. The result of the significance test is expressed as a probability that measures how much the data and claim agree. In simple terms, tests of significance help you determine whether an observed effect is real or a fluke of the sample. Some significance tests include Z-test, T-test, Chi-square, F-test/ANOVA, Correlation test, etc.

Broadly, significance tests are categorized into the following two tests:

1. Parametric methods

Parametric Tests assume specific distributional forms (often normal). These include z-test, t-test, Chi-Square test (χ²-test), and F-test/ANOVA. Such tests of significance have greater statistical power when these assumptions hold.  Let us discuss different parametric methods:

• t-test: It can be of three types including one-sample t-test, independent two-sample t-test and paired t-test.
• Z-Test: This test of significance is used when population variance is known or if the sample size is large and the data are approximately normal.
• Chi-Square (χ²) Test: It compares the observed categorical frequencies to expected frequencies under the null hypothesis. It has the following two applications:
  • Goodness-of-fit: This tests whether observed categorical frequencies are matching the expected proportions
  • Test of independence: It assesses association between the two categorical variables.
• F-Test & ANOVA: While F-test compares the variances of 2 populations, one-way ANOVA tests mean differences across 3 or more groups.

2. Non-parametric method

Non-parametric tests require fewer assumptions about the data. This type of significance testing include Wilcoxon signed-rank, Kruskal–Wallis H Test and Mann-Whitney U test. These are suitable for ordinal/non-normal data and small samples.

• Mann-Whitney U test: This test is an alternative to the two-sample t-test for non-normal/ordinal data.
• Wilcoxon Signed-Rank Test: It is an alternative to paired t-tes which compares two related samples or related measurements.
• Kruskal–Wallis H Test: This a non-parametric analog of one-way ANOVA which compares ranks across 3 or more groups.

A statistical significance test is an important procedure for the following reasons. 

• These tests allow researchers to determine if the observed effect in the sample data reflects genuine phenomena in a broader population or is due to random variation.
• Tests of significance in statistics help in decision making by quantifying the likelihood of observing data under the null hypothesis. This controls the risk of false positives. 
• Without significance testing, it is not possible to distinguish meaningful patterns from chance fluctuations. 
• GATE (Statistics), CSIR NET (Mathematical Sciences), and IIT JAM (Mathematical Statistics) all cover questions related to hypothesis testing. 
• Drawing valid inference from samples to populations is not possible without significance testing.
• Significance tests quantify how likely a sample result could occur if there is no effect in the population. This fills up the gap between limited data and broader conclusions.
• Tests of significance can provide a statistical basis to determine whether observed changes are exceeding what random noise would produce on its own.
• By setting the significance level (α), researchers can limit the probability of incorrectly rejecting the true null hypothesis (Type 1 error). 
• Test of significance in statistics converts data into a p-value, which is the probability of observing equally or more extreme results if the null hypothesis is true.  A lower p-value shows stronger evidence against the null.

The following stages are involved in a test of significance:

Step 1: Creating a research hypothesis

Whenever you start with a null hypothesis with “no effect” or “no difference”, you posit an alternative hypothesis. The alternative hypothesis means what you actually suspect. A test of significance will determine how compatible your data is with H₀.

Let us consider an example to explain how to create a research hypothesis: “Does a 30-minute daily brisk walk reduce resting systolic BP in 40–60 year-olds with sedentary lifestyle over a period of 8 weeks?”

Here, the population is sedentary adults in the 40-60 age group. Intervention is a 30-minute daily brisk walk for a period of 8 weeks

The outcome is a change in the resting systolic blood pressure (mmHg)

Step 2: Identifying Variables

The other aim is to identify independent and dependent variables.

1. Independent Variable (IV)

Here, the independent variable is walking regime. It has two levels.

First is the walk group, which is a 30-minute brisk walk every day.

Second is the control group, which maintains the usual lifestyle, i.e. sedentary lifestyle every day.

2. Dependent variable (DV)

It is the resting systolic BP (mmHg) which is measured before and after 8 weeks.

Step 3: Formulating the Hypothesis

The third step in most of the tests of significance is hypothesis formulation. First, let us consider the Null hypothesis (H₀).

As per this, there is no difference in the mean change of resting systolic BP between the walk and control groups. $$H_0:{\mu }_{\text{ΔBP, walk}}={\mu }_{\text{ΔBP, control}}$$ Now, let us consider the alternative hypothesis. As per this hypothesis, daily brisk walking decreases the resting systolic BP as compared to control. Here, we have pickeda one-tailed test since prior studies suggest that exercise lowers your BP. $$H_1:{\mu }_{\text{ΔBP, walk}}<{\mu }_{\text{ΔBP, control}}$$

Step 4: Significance Level and Plan Sample Size

α = 0.05 (5% chance of falsely “finding” an effect)

Power ≥ 0.8 → Estimate requires ~30 participants per group (based on the expected SD and effect size)

Step 5: Data Collection and Computing Test Statistics

• The first step in this part is to measure the baseline BP of every participant. Assign randomly to the walk vs. control. After a period of 8 weeks, measure the resting BP again.
• Compute the difference between BP, i.e. ΔBP = (Post – Pre) for every participant.
• Use an unpaired t-test (since these two are independent groups). Had the groups been related, you should have used a paired t-test. For two independent samples (walk vs. control), the t-statistic is:

$$t=\frac{{\bar{x}}_{\text{walk}}-{\bar{x}}_{\text{control}}}{s_{\text{p}}\sqrt{\frac{1}{n}+\frac{1}{n}}}$$

Step 6: Find the p-value and make your call

Say you get t = -2.15 with df = 58. This will yield p = 0.018 (one-tailed).

Since p (0.018) < α (0.05), you will reject H₀.

Step 7: Reflect Critically

The last step in most tests of significance is considering the effect size. Here, a 5mmHg drop can be clinically important depending on the context.

Further assumptions that must be checked are:

1. Is the ΔBP distribution roughly normal?
2. Are there any dropout biases?
3. Have both groups maintained similar salt intake?

1. whether the new machine reads higher or lower.

Students who are planning to take board exams soon, must check out the following pages as well for thorough preparation:

Do remember that hypothesis tests are based on incomplete information, which means a sample can never provide complete information about a population. It is therefore important to note that there is always a chance that conclusions have been made in error. Now, you must know the types of errors that are possible here. CBSE board exam can ask question directly about the error types in significance test. Let us discuss them one by one. 

• Type I error: This is the first possible error in tests of significance. It occurs if we conclude that null hypothesis is invalid even when null hypothesis is actually true. This is known as the Type 1 error.
• Type II error: If we conclude that null hypothesis is reasonable while null hypothesis is actually false. This is a case of Type 2 error.

In terms of significance of hypothesis testing, p-value is a function that quantifies how extreme your observed data is under the assumption that null hypothesis is true. The p-value is probability of obtaining a test statistic at least as extreme as the one observed, if there is actually no real effect or difference in the population. Smaller p-values indicate that such an extreme likely is unlikely under null, thus providing stronger evidence.

Under the null hypothesis H₀, the p-value is:

$$p=\mathrm{Pr}\left(T\ge t\right)|H_0$$

This is for a right tailed test where T represents the test statistic and t is the observed value. Here, it is assumed that null hypothesis $H_0$ is true.